BiLD (Bi-weekly DIRAC Development meeting) – 30/07/2020
At CERN: Nobody, of course!
On Zoom: Federico, Andrii, Andrei, André, Alexandre, Aymane, Cedric, Christopher, Daniela, Hideki, Janusz, Marko, Xiaomei, Raja, Ruslan (Belle2), Simon
Apologies:
Follow-up from previous meeting
- DIRAC releases on CVMFS:
- Andrei setup a nightly cron job which is updating the CVMFS with all those that are in releases.cfg
- Also removes from CVMFS those that are removed (still a lot of clean-up to do)
- Also updates the global cfg file, the tarballs, and JSON for pilot
- Keeping it in local for now, but should be put in GitHub Actions, and then use secrets
- Marko: I will prepare you a container in management repo
- TLS issue: https://github.com/DIRACGrid/DIRAC/issues/4675 has been closed (should be solved in modern versions)
- Error in Pilot related to locale (https://github.com/DIRACGrid/Pilot/issues/112). Deployed: can it be confirmed that fixes the original issue by Nica?
DIRAC communities roundtable
GridPP:
- Still running v6r22
- Pilot logging: We are still trying to get our heads around that one.
- Pull requests in preparation for v7r1
- cloudinit VMDirac: Needed for Pilot3
- Pull requests in response to user issues:
- https://github.com/DIRACGrid/DIRAC/pull/4695 (Fix MutiVO Catalog tests) - this looks done
CLIC:
LHCb:
- Running certification using v7r1. Looks OK, just not much time now for production.
France Grilles:
- CC/Lyon started providing pilot of ES service. One peculiarity is that it needs logging in with certificate. Need DIRAC patch for using it.
EGI:
- Setting up token-based AuthN server side. Still discussions going on.
Belle2:
- Migration to v6r22: not yet, because of local maintenance.
- Faced one problem: attempted to use latest lcgBundle (r11)
- Problem in RMS when using Davix protocol - backported the version
- AT: why RMS? - HM: thread never answering (gfal2 library issue?)
- DIRACOS should be of course the solution, but not certified for davix.
Nica:
Juno:
Current situation
DIRAC
- v6r22:
- v6r22p32
- ElasticSearchDB: backported update() function from v7r0.
- ARCComputingElement - reserve the number of cores as defined in the NumberOfProcessors configuration parameter
- v7r0:
- v7r0p29
- M2Crypto fixes (not final)
- Configuration: add --cfg option for config files, deprecating old non-option version.
- Few fixes for TS and PS
- Hack in Slurm to stop jobs running out of time on multi-processors resources
- v7r1:
- v7r1p7
- dirac-install is getting the globalDefaults.cfg from CVMFS if available, loads the installation configuration from etc/dirac.cfg if it is present.
- NEW command dirac-install-extension - new command to add an extension to an existing installation
- WMS: DownloadInputData no longer fails if getFileMetadata fails for the first SE, tries others if available
- v7r2:
WebApp:
- New issues:
- A few PRs to review + merge
Pilot3:
DIRACOS:
- xroot5?
- Last version includes the newest version of M2Crypto
- RN: why the
less
command doesn’t work? - MP: it’s a known bug. It’s a bit too much work to fix it, vi
works. AT: just don’t source bashrc… not many solutions.
VM:
Documentation:
OAuth2:
tornado and other externals
- M2Crypto: a PR for DIRAC is in WIP
management
diraccfg
Release planning, tests and certification
- Big PRs for v7r2 are not yet reviewed. Will need to wait for them before really starting certifications.
Weekly development(s) focus
The discussion was on 3 points:
- how to implement REST in DIRAC. 2 options, right now:
- use “Chris PR”
- use current WebApp framework
AT: the 2 implementation are just different. We need to sort out how we want to go in the future. Will open an RFC for discussion.
- base library used for implementing the OAuth protocol. 2 options:
- use an existing one (e.g. https://github.com/oauthlib/oauthlib or https://github.com/lepture/authlib)
- create our one – this is what’s done right now
Chris B will share his experience with Andrii about using the existing external ones
- semantic of components
FS: why VOMS is not part of Resource? - AT: we don’t have ATM a way to define its semantics, maybe we need yet another Resource type. - FS: we will leave for long time in a mixed environment, we need clear definitions. - AT: I will create an RFC for discussion.
DIRAC: current PRs and tasks being worked on, or topics from Google forum
PRs (there was no time for further discussions):
On issues:
Topics from Google forum:
- ElasticSearch setup:
- How to reset files in “processed” status: question from Xiaomei – …?
- long jobs - proxy renewal: question for ARC and FG setup
- Stage out retry and how do I test this: generated one issue https://github.com/DIRACGrid/DIRAC/issues/4694 (see above)
- API::job submission:: specify the number of replicas (and SEs) for output: basically a feature request, do we tranform it into a GitHub issue?
AOB
CERN policies still forbid travel. Can’t know yet when there will be next DUW.
EGI workshop in November will go fully virtual. 1 day DIRAC session? probably - AT in contact.
Next BiLD in 2 weeks. There will be a 5 weeks with no BiLD after that.
LHCbDIRAC