BiLD (Bi-weekly DIRAC Development meeting) – 10/06/2021

At CERN: Still, nobody :/
On Zoom: Federico, André, Andrei, Andrii, Cedric, Christopher, Daniela, Hideki, Janusz, Marko, Simon, Yuji, Vladimir
Apologies:

Follow-up from previous meetings

• Ran hackathon last week on 7.3.0a10:
  • trello board here
  • didn’t go very far as there were issues with Jobs Monitoring that occupied most of the debugging time
• PRs merging strategy?
  • this is done, but some discussions emerged so not yet applied (see PR)
  •  Federico did not yet try in DIRACGrid-test.
    •  Christopher I did try, it works well and smoothly
• Poll on using black or not: https://docs.google.com/forms/d/1c0B4l5AfZTg8UZXG4DPQw_OZ59YOTYd2leesnYiPhIY/
  • 8 answers for now, will be disclosed at the next meeting

DIRAC communities roundtable

GridPP:

 Daniela

• Updates in https://github.com/xrootd/xrootd/issues/1198 (EPoll: Bad file descriptor polling for events)
  • solution on sight (will need a new release of xrootd)
    •  Federico+Christopher DIRACOS has anyway not updated to version 5 so probably this will be much later for us
  • env variable removal looks like working fine in any case
• User request: Could “listCatalog” that is currently commented out in the DIRAC API please be enabled ? (Why do I get the feeling that the answer will be: Sure, can you do it ?)
  •  Andrei implemented in the console. Create an issue, assign it on me.
• We haven’t had any more input from the user about the xrootd 3rd party copy problem, I assume they are all on holiday.

CLIC:

 André

• Started looking at v7r2, migration end of July
  • Trying first to get the python2 tests to work, will look at the python3 client later

LHCb:

 Federico+Chris+Chris+Marko

• Deployed v7r2 in production
  • main issues was an incompatibility with v7r1 fixed in PR
  • following the deploy, other issues were investigated, mostly performance-related. Some PRs created by Christopher and Federico
    •  Christopher all my PRs include fixes that are not specifically depending from this release
• Sandbox store (re-iterated) suggestion: use a high performance IO volume (LHCb is a bit struggling atm with cloud-based not-so-performant volume)
  • FYI: Available storage types at CERN: https://clouddocs.web.cern.ch/details/block_volumes.html. We were previouly using the “standard” type but the 100 iops is a bit too low for LHCb
• py3 client: not advertised yet, a few issues to be sorted out first (connected to LHCb specificities)

EGI (+ France Grilles):

 Andrei

• from previous meeting M2Crypto and VOMS extension follow-up (issue)
  • Robot certificates regenerated once again, still to be tested
• v7r2 deployment on several nodes for testing (EGI has no DIRAC extension, but some users do)
• Started to use ES (certificate access), WMSHistory is enabled
  • very slow to connect, looking with local manager what’s the issue
  • still, working. So enjoing WMSHistory on ES7.

Belle2

 Hideki

• Postponed the migration to v7r0, issue should be sorted and next week deploy hopefully.

IHEP

 Xiaomei

• On the way to migrate to v7r1. Plan to use Pilot3
• CS and WebApp on same machine, upload necessary?
  • it shouldn’t be needed to have 2 different machine
•  Federico It’s possible to get the Pilot files not from github (which is the default)
• Some users used the registerFile (from CLI), without checksum … this becomes problematic, does it need to be there?
  •  Daniela we had that problem before. We gave some code to do it programmatically “correctly”. Best solution would probably be to reject registrations without checksum, while adding a “force” flag just in case.
  •  Andrei this is a admin operation, or should be
  • Please create an issue on github!

DIRAC releases

• v7r1p41 + v7r1p42:
  • Bugfixes (UserProfileDB, PilotSubmission accounting)
  • Performance improvements for Framework.Monitoring and Public State Manager
• v7r2p8 + v7r2p9:
  • NEW: (#5122) multi-node allocation support via the ParallelLibraries and srun (slurm)
  • NEW: (#5155) ComponentSupervisionAgent to Monitor running components.
    • for executors careful: might not working if all the executors are together, or on different machines
    • https services: maybe not working, because not in SystemAdministrator (https://github.com/DIRACGrid/DIRAC/issues/5188)
• v7r3-pre10 (7.3.0a10):
  • CHANGE: (#4715) removing env variable DIRAC_USE_NEWTHREADPOOL (use concurrent.futures instead of DIRAC ThreadPool for agents)

DIRAC projects

DIRAC:

Issues by milestone:

• v7r1:

  • some issues have been closed

• v7r2:

  • not much done

• v7r3:

  • many issues are open. Some might go to v7r4

• What we want from the plotting

  • “Hard deadline for comments: 15th of June”

PRs discussed:

• from previous meeting [v7r2] Helper functions to make working with errors easier
  • several comments to it, deserves some discussion, but no updates
• Several PRs created by  Christopher for v7r2 that aim at performance fixes. The only one that might be slightly dangerous is [v7r2] Add environment variable to speed up the CS
  • Connected to https://github.com/DIRACGrid/diraccfg/pull/5
  • the suggestion for this one is to test it for a couple patches in LHCb and then make the new way the default

WebApp:

•  Federico add VMDIRAC app?
  • https://github.com/DIRACGrid/VMDIRAC/issues/152
  •  Andrei the best would be to just merge VMDIRAC into DIRAC
    •  Simon that was always the plan
    • the JavaScript part should be brought app to the latest version
      • opened issue for that
•  Janusz working on a WebApp for RSS
•  Andrii In hackathon we found issue for encoding JSON, so this should be fixed before the next hackathon.

Pilot3:

• NTR

DIRACOS:

• xroot5?
  • NTR
• condor 9?
  • from previous meeting Overall, HTCondor 9.0 brings many new features but most of them are probably not really useful for DIRAC.
  • no news on integration in DIRACOS (done for DIRACOS2)

DIRACOS2:

• from previous meetingfts-rest on py3?
  • NTR
• from previous meeting Question for rucio client… (issue)?
  • NTR
• added cachetools
• oidc-agent? issue opened by  Andrii
  • at the moment not needed, maybe it will be needed in the future. For the moment the issue can be closed.

VMDIRAC:

• Agreed to merge into DIRAC for v7r3
  • should be rather easy.  Simon will take care
• For the Web part a bit more work is necessary.

Documentation:

• 1 draft PR open, will be expanded with py3 documentation.

OAuth2:

• from previous meeting Any news from issue WLCG and Token transition – reminders and requirements ?
  • hackathon last week
    • some practice on already implemented tools. HTCondor providing access policies with tokens. Doc provided and discussed on how pilots will use tokens on the CEs. Notes from the hackaton
• from previous meeting  Andrii forwarded https://docs.google.com/document/d/1ZQzElD866yV6t9AomeW6r-L9s-64pNB95OFDd1EZEbA/edit?ts=60814a09#heading=h.ur0csvxkk8tc which contains technical drawings about the proposed implementation.
  • anyone can comment on it as of now. Please do read it!
• dirac-login will (one day) end up superseeding dirac-proxy-init
• python2 compatibility: we will drop support for this development
• There are questions on the https implementation found in the current PR, but the point has not been expanded in the meeting

tornado/HTTPs

• NTR

management

• from previous meeting 1 task left for deploying on CVMFS
  • Not yet done  Marko I will come back to it

COMDIRAC

• NTR

other externals, including Rucio

•  Cedric No progress, tests still to implement/to pass

  •  Federico will contact Janusz and Cedric for technical explanation

•  Janusz working on Cedric’s code for:

  • making it multi-VO
    • doesn’t look very complex, at least for what regards the SynchronizationAgent
      • testing with 2 VOs in GridPP setup
    • RucioRSSAgent looks a bit more complex (will come next)
  • eliminating the need for Rucio config
    • a few changes to the rucio client itself are needed

Release planning, tests and certification

• Certification machine(s)

  • authcertif machine:
    • py3 server installation being tested (the installation procedure is OK)
    • the client will also need to be py3 with a specific code

• Next hackathon

  • “regular” next week
  • “auth” later (first need to actually fully install it)

Weekly development(s) focus

NTR

AOB and topics from Google forum

Next hackathon (regular) on June 17th: https://indico.cern.ch/event/1048251/
Next BiLD on June 24th: https://indico.cern.ch/event/1048250/

LHCbDIRAC

• https://trello.com/b/BQvEbteg/deploy-v10r2
• python3 client on CVMFS
  • Ganga does not work
  • LHCbPilot would need to add a flag
  1. get Ganga to work first
  2. prod for “free users”
  3. for LHCbPilot, we can have a flag and then enable for a subset of the pilots that we send or run
• SandboxStore: running in certification, looks fine, should be installed in prod