BiLD (Bi-weekly DIRAC Development meeting) – 07/04/2021

At CERN: Still, nobody :/
On Zoom: Federico, Alexandre, André, Andrei, Andrii, Christophe, Christopher, Daniela, Hideki, Janusz, Marko, Simon
Apologies:

Follow-up from previous meeting

• Ran hackathon last week on v7r3-pre3:

  • trello board here
  • VMDIRAC is installed and gave a few issues, but we didn’t properly tried it
    • e.g. taking dirac-resource-get-parameters.py from VMDIRAC, giving few issues
      • right now the pilots do dirac-install -e VMDIRAC which is wrong in grid context.
      • Q: if it’s taken from CVMFS, might pose issues…
    • Simon+Daniela made an update yesterday, now at least VM get started, to be debugged.

• PRs merging strategy?

  • “Sweeping” is the name that was used by ATLAS in GitLab for doing the action
  •  Marko took that, and started adapting it to GitHub: https://github.com/petricm/sweeper/
  • Testing: https://github.com/DIRACGrid-test/DIRAC/pulls
    • a few things need to be re-written for GitHub.
  • As of now, new PRs for other branches are created after the first PR is merged.

DIRAC communities roundtable

GridPP:

 Daniela

• NTR

CLIC:

 André

• Fixed some issues with VOMS2CSAgent and the same users in multiple VOs (v7r0)

LHCb:

 Federico+Chris+Chris+Marko

• Singularity as InnerCE
  • activated some weeks ago on specific CEs (mostly T1s), now on few T2s. No issues on our code but somewhere it just can’t work (e.g. missing some CVMFS locations). We need a better monitoring as errors are hard to spot (jobs get rescheduled).

EGI + France Grilles:

 Andrei

• Fixed problems with robot certificates. M2Crypto is still disabled but should work now, as tests are OK.
• Still doing a lot of cleaning on Resources
• arc client: might be needed.
  • to try use --dirac-os-version=arc6
  •  André Try enable debug for ARC https://github.com/DIRACGrid/DIRAC/blob/f20575f871aae83df969d494e4cbb97651c3e70f/src/DIRAC/Resources/Computing/ARCComputingElement.py#L29-L34

Belle2

 Hideki

• Started migration to v7r0. Planned for end of April or May.

DIRAC releases

• v7r1p35 + v7r1p36:
  • NEW: (#5047) DIRAC_DEBUG_M2CRYPTO env variable for SSL debugging
  • CHANGE: (#5053) Remove margins from the TimeLeft utility
  • CHANGE (#5051) StalledJobAgent fixes
  • FIX: (#5080) SQLAlchemy 1.4 support
• v7r2p1 + v7r2p2:
  • NEW: (#5057) Allows to retrieve the non recursive size of a directory from the DFC
  • CHANGE: (#5070) StalledJobAgent is now threaded
  • python3 fixes
• v7r3-pre3:
  • NTR

DIRAC projects

DIRAC:

Issues by milestone:

• v7r1:
  • no updates
• v7r2:
  • NTR
• v7r3:
  • basically almost everything needs to be done.

PRs discussed:

• python3 PRs: almost ready to go
• RSS multi-VO: almost ready to go

WebApp:

• v4r3 tag is to be used for v7r3
• fixed bug on updating images.
• working on OAuth PR

Pilot3:

• https://github.com/DIRACGrid/Pilot/pull/129
  • added a switch to pilot for installing a python3 DIRAC
  • no updates

DIRACOS:

• xroot5? (last issue there)
  • to be done, still. It’s in DIRACOS2.
  • no much time
• arc6
  • almost there
• condor 8.8
  •  Marko trying it, there are a few additional dependencies.
• Issues downloading DIRACOS on slow connections as part of the integration tests
  •  Christopher You can set DIRACOS_TARBALL_PATH to copy DIRACOS from a local file

DIRACOS2:

• from previous meetingfts-rest on py3?
  • NTR
• from previous meeting Question for rucio client… (issue)?
  • NTR
• Linking https://github.com/DIRACGrid/DIRACOS2/issues/30

VMDIRAC:

• No issues nor PRs

Documentation:

• NTR

OAuth2:

• from previous meeting Any news from issue WLCG and Token transition – reminders and requirements ?
  • NNTR
• PRs:
  • DIRAC: https://github.com/DIRACGrid/DIRAC/pull/5045
  • Web: https://github.com/DIRACGrid/WebAppDIRAC/pull/447

tornado/HTTPs

• NTR

management

• from previous meeting only one task left for deploying on CVMFS
  • updates?
• created issue on python3 setup: deployment of DIRACOS2 and pip-installable packages after last meeting
• Issue: Where should etc/ be for Python 3 based installations?

COMDIRAC

• NTR

other externals, including Rucio

• Old PR (that was targeting v6r22) moved to targeting v7r0: https://github.com/DIRACGrid/DIRAC/pull/5067
  • failing tests
  •  Janusz I tried integrating in our setup. Didn’t try the agents.
    • configuration: Rucio insists on having a config file, but all these configs should (could) be put in DIRAC CS. We should insist on Rucio to not needing this config file.

Release planning, tests and certification

• Certification machine

  • VMDIRAC
    • Getting there
  • Reminder: task for multi-vo

• Next hackathon

  • next week (15th April)
    • v7r3-preX

Weekly development(s) focus

On OAuth

• Asked DIRAC instances admins if they looked into the topic
  • mostly, not
• Some discussions from meeting with few devs, that happened a few days ago: https://demo.hedgedoc.org/38mzdSQpT36MifxBp7xX8g
  • anyone is welcome to read and participate to it
• In general, at some point each and every VO of DIRAC will need to move to interact with IAM or Check-In.
• For testing, it would be good to have a development instance. dteam and lhcb endpoints would be needed.
•  Andrei another technical thing that can be tried in parallel is to try to use the services that accepts tokens (independently of other changes in DIRAC)

Links:
* Diagrams of interactions added in https://github.com/DIRACGrid/DIRAC/pull/5045#issuecomment-811554323
* Issue + discussions on REST in OAuth: https://github.com/DIRACGrid/DIRAC/issues/4703

AOB and topics from Google forum

Next hackathon on April 15th: https://indico.cern.ch/event/1026974/
Next BiLD on April 22nd: https://indico.cern.ch/event/1026975/

LHCbDIRAC

• v10r2 planning in https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/wikis/Releases-planning#based-on-dirac-v7r2
• https://codimd.web.cern.ch/2es8HPUcSmS42jrTxhX3JQ?both
  • MySQL 8:
    • should be done
• https://trello.com/b/ECkoWbmk/deploy-v10r1
• At some points we should install a python3 client on CVMFS
  • in the hands of lb-dirac (then LHCbPilot would need to add a flag)