BiLD (Bi-weekly DIRAC Development meeting) – 10/08/2023
At CERN: Christophe, Vladimir
On Zoom: Federico, Bertrand, Christopher, Simon F, Ueda, Janusz, Lorenzo, Kushagra Singhm, Ewoud, Xiaomei, Alexey, Simon M
Apologies: Daniela, Andrei
Follow-up from previous meetings
- Last “standard” BiLD 2 weeks ago
- Trying to catch all updates below
- Last DIRAC certification hackathon on August 3rd
DIRAC communities roundtable
LHCb:
Federico+Simon+Christophe+Christopher
- Running on latest v8.0 and latest DIRACOS2 versions
- Using tokens for submitting to almost all HTCondorCEs and some AREX
- Latest v8 WebApp was problematic for LHCb, and had to remove the “Tabs” from cfg.
CLIC/ILC/Calice
Lorenzo
Belle2
Ueda, Kushagra
- v7r3 py3 on servers – py3 for clients with some exceptions
- pilot still uses py2 – for setting up DIRAC, not installing it
- preparation for DIRAC v8 ongoing. Right now some issues while adapting WebApp.
- from previous meeting DIRACOS2 migrated to py3.11, showed incompatibility with one of the external services.
Juno/BES3
Xiaomei
- Moving to v8.0
- Ran into py3.11 issue with https://github.com/DIRACGrid/DIRAC/discussions/7165
- Christophe please put new error in the ticket
- Federico Seems more like an
openssl error rather than python version- and ideally try to reproduce the error with pure
openssl command
- ticket solved later in the day
GridPP:
Simon, Janusz
- No updates to the production server. (No v8.0 relase with ARC-CE fixes yet ?)
- Tested Chris H’s SingularityCE fixes on pre-prod, seems to work.
- Still working on TransformationSystem for v8.0 for improved security model, waiting for input from Luisa wrt Production system: [#7113] (https://github.com/DIRACGrid/DIRAC/pull/7113)
- For integration, we will port the code using username instead of userDN now that #7124 is resolved.
- We could really do with a working certification server because otherwise we can’t test it, Daniela hasn’t managed to run a job on integration for months now, grumble.
- No news on tokens (i.e. we haevn’t done any further work on this): #7123 (also: #7126)
- Daniela & Simon have now booked their flights to Japan :-)
Topics from GitHub/Discussions
only un-answered topics with discussion updates:
DIRAC releases
- v7r3
- Latest patch incorporates all fixes
- v8r0
- v8r1
DIRAC projects
DIRAC:
Issues by milestone:
v8.0:
v8.1:
Other issues:
PRs discussed:
WebApp:
Pilot:
- Federico general suggestion for development on Pilot code in order to avoid “overconfident” merges of PRs to
master branch, as experience has shown that it is often delicate. They would also make for a simpler workflow.:- align
master with devel (~now) - forbid opening of PRs to
master branch - always test
devel in DIRAC certification setup (and in github Actions and in Jenkins) - (more) frequently merge
devel to master
- Discussion
- Daniela (in absentia): This only works if it is actually possible to run jobs in certification.
- Christopher if we have many changes in
devel it will be difficult/risky to merge- Federico I thought about having also another branch, e.g.
releaseCandidate branch, but effectively we can always test in production also contents of PRs before merging them
- Chistopher can we think at a better way of testing this? Maybe submitting partly
devel and partly master pilots. - Federico points 1 and 2 above are probably OK anyway. We can wait for the next BiLD to see if there is any other suggestion
- PRs:
DIRACOS:
- Christopher No longer installing Singularity and Apptainer, just Apptainer (with simlink to singularity)
Documentation:
Asking to Add documentation on how to write scripts that interact with DIRAC
mail from rtd:
We are removing the “use system packages” feature on August 29th. Make sure you are installing all the required dependecies to build your project’s documentation using a requirements.txt file and specifying it in your .readthedocs.yaml.
Here you have an example of the section required on the .readthedocs.yaml configuration file:
python:
install:
- requirements: docs/requirements.txt
- Christopher we’re using the Conda env, so nothing to do.
OAuth2:
- from previous meeting
- Andrei request from EGI to demonstrate that one VO can run with tokens only
- Check In is progressing:
compute scopes available, they are accepting the idea of using client access tokens (possibility to associate a client to a given VO). They would probably not accept a same client to deal both with client and user access tokens (security concerns with the scopes available in the clients). - WLCG timeline document: https://zenodo.org/record/7014668#.YyLag9JBwQ9
- Still pending the test for ARC7 and CheckIN tokens
- Multiple clients per IdP might be effectively needed
- Asked about multi-VO setup and token tags
management
- from previous meeting 3 issues left, still valid?
- Always upload releases to CVMFS is again valid for clients
- higher priority now. This should also be done for non-x86 versions.
diraccfg
DB12
Alexandre
Rucio
Tests
DiracX
- from previous meeting
- Called a 2nd DiracX hackathon: https://indico.cern.ch/event/1304626/ for 4-5 September. Again at CERN, please do register if you plan to attend.
- Also called a BildX meeting for August 31st (the week before the hackathon)
Release planning, tests and certification
Certification machines
- lbcertifdirac70 machine:
- from previous meeting Federico not rush, but should we move to a Alma9 box?
- Outside of CERN would be better, in CC probably
- Andrei machine is already there, need to decide how to set this one up
- We could also use the new box to test the installation procedure
Next hackathon(s)
- in 2 weeks, “standard” v8.1.0aX one.
AOB
Next hackathon: August 24th
Next BiLD: September 14th (no possibility before)
LHCbDIRAC
v11.0.17 and v11.0.18
- Created v11.0.17 a bit in a hurry (DPA pushing) but changes for
conditions_id impacted other parts. Decided to revert the specific PRs and create a new release.
v11.0: deploy board in https://trello.com/b/Ep0PAkbv/deploy-110
LHCbDIRAC hackathon?
- 17th Aug. Will be mostly for restoring the setup and verify if there’s anything dramatically broken.