BiLD-Dev
Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.
Join Zoom Meeting
https://cern.zoom.us/j/91083694183?pwd=ZkdDY1I1YkJVc2o3UTdBY1BRZE15UT09
Meeting ID: 910 8369 4183
Passcode: 12345678
One tap mobile
+41432107108,,91083694183# Switzerland
+41315280988,,91083694183# Switzerland
Dial by your location
+41 43 210 71 08 Switzerland
+41 31 528 09 88 Switzerland
+41 43 210 70 42 Switzerland
+33 1 8699 5831 France
+33 1 7037 2246 France
+33 1 7037 9729 France
Meeting ID: 910 8369 4183
Find your local number: https://cern.zoom.us/u/artUfAdNB
Join by SIP
91083694183@188.184.89.188
91083694183@188.185.118.153
Join by H.323
188.184.89.188
188.185.118.153
Meeting ID: 910 8369 4183
Passcode: 12345678
BiLD (Bi-weekly DIRAC Development meeting) – 24/05/2023
On Zoom: Federico, Andrei, Alexey, Alexandre, Alexey, André, Bertrand, Daniela, Ewoud, Igor, Simon F, Simon M, Ueda, Hideki, Michel
Apologies:
Follow-up from previous meetings
- Last BiLD many weeks ago
- Easter, then CHEP in the meanwhile
- Last hackathon long time ago
DIRAC communities roundtable
LHCb:
Federico+Alexandre+Simon+Christophe+Christopher+Alexey
- Installed latest DIRAC production release (v8.0.20), which was actually buggy (SiteDirector). Fix PR already created and merged.
- Got a request from DESY on moving to HTCondor 10 (only tokens)
EGI
Andrei
- Running services smoothly. Few hotpatches to allow using tokens in EGI. Yesterday managed to submit pilots using tokens from CheckIn.
- Igor does EGI accounting work correctly with tokens?
- Andrei several discussions ongoing on this topic.
CLIC/ILC/Calice
André
- Moved the main servers to v7r3, py3
Belle2
Hideki, Michel, Ueda
- v7r3 with python2, trying to migrate to python3
- Discussing about submitting HTCondor using tokens. Andrei suggests to run just the SiteDirectors to v8
GridPP:
Daniela, Simon
- No change in production, running v7.3.36. Strongly suggested to our possibly last remaining python2 DIRAC UI user that time might be running out.
JINR
Igor
- Pilots rather “large”, basically pointing to https://github.com/DIRACGrid/Pilot/issues/166
Topics from GitHub/Discussions
only un-answered topics with discussion updates:
- OSError(16, ‘Device or resource busy’) in Parameters of a DIRAC job
- AgentMonitoring and JobAgent entries in ES
- I guess it’s a bug…
- Removing the possibility for custom getDIRACPlatform(s) method ?
- Validate the DIRAC Job format (JDL) using Pydantic
- About RAL setup, old xroot version and XrdSecGSIDELEGPROXY variable
- https services questions
DIRAC releases
- v7r3
- Last one is v7.3.
*
- Last one is v7.3.
- v8r0
- Last one is v8.0.20
- Buggy! New one will be created ASAP
- Last one is v8.0.20
- v8r1
*
DIRAC projects
DIRAC:
Issues by milestone:
v7r3:
- Only one issue left, about documentation. Christophe do you still want to take care?
v8.0:
- 20+ open issues
- Mix of groups for ReportGeneratorHandler
- seen in LHCb production
- Mix of groups for ReportGeneratorHandler
v8.1:
- 21 open issues
*
PRs discussed:
- [8.0] Add dirac-admin-add-pilot command
- Connected to Pilot PR master: add new command RegisterPilot
- [8.0] Separate handler class per service
- Should be OK to merge
- [8.0] print a warning message when no CFG file is found
- Please review once more!
- [8.0] feat: use client token instead of user token for submission
- merge now? Seems yes.
- Discussion about the structure in CS should be implemented in a different PR
- [8.0] fix the interactions between the Matcher and the PoolCE
- One more update needed before merging this one
- SudoCE can be dropped
- [8.0] remove elastic job parameters when removing jobs
- Maybe go through index policy
- [8.1] feat: RequestManagementSystem: remove DN in favour of owner
- Ready to be merged (since a while). In absence of a review, I will merge this one on May 30th
- [8.1] integration tests: HTTPs services by default
- Ready to be merged (since a while). In absence of a review, I will merge this one on May 30th
- [8.1] Parsing the jdl input by using the Job API
- [8.1] Validating the JDL format using pydantic
WebApp:
- NTR
Pilot:
DIRACOS2:
- Files not found when sourcing diracosrc with latest diracos release
- Updating DIRACOS2 to OpenSSL 3
- from previous meeting Python 3.11 seems to be fine
- Propose to update immediately after OpenSSL 3
Documentation:
- NTR
OAuth2:
Andrei request from EGI to demonstrate that one VO can run with tokens only
The existing DIRAC Authorization Server already takes care of refreshing the tokens. This is built in but the DIRAC groups are not re-dded correctly.
from previous meeting Check In is progressing: compute scopes available, they are accepting the idea of using client access tokens (possibility to associate a client to a given VO). They would probably not accept a same client to deal both with client and user access tokens (security concerns with the scopes available in the clients).
from previous meeting
- WLCG timeline document: https://zenodo.org/record/7014668#.YyLag9JBwQ9
- Meeting note from 8/12/2022: https://demo.hedgedoc.org/cWl2Y5MtTwSn4V5Uic_K5w
tornado/HTTPs
- Streaming solution still needs to be properly thought.
- from previous meeting Issue https://github.com/DIRACGrid/DIRAC/issues/6495 keeps track of what can and what will not be moved to https
- NTR
- from previous meeting Federico we need to try and propose a coherent update from DIPS to HTTPS services, e.g. for Configuration Services URLs
management
- from previous meeting 3 issues left, still valid
diraccfg
- version 1.0? still tbd
COMDIRAC
Daniela
- We are slowly but surely addressing the points listed in: https://github.com/DIRACGrid/DIRAC/issues/6746
- This meeting’s pull requests: https://github.com/DIRACGrid/DIRAC/pull/7023 (documentation) and https://github.com/DIRACGrid/DIRAC/pull/7024 (hackathon tests)
DB12
Alexandre
- NTR
Rucio
- NTR
Tests
- Moving to HTTPs services as defaults in PR
- discussed above
Release planning, tests and certification
Certification machines
- lbcertifdirac70 machine:
- NTR
- lbcertifdiracoauth machine has been destroyed
- Federico not rush, but should we move to a Alma9 box?
- Outside of CERN would be better, in CC probably
- We could also use the new box to test the installation procedure (now we have mostly HTTPs services)
- ?? Alternatively, or together with it, it would be nice to have a K8 setup
Next hackathon(s)
*
AOB
- Post CHEP topics of interest:
- Nothing for today
Next hackathon: TBD
Next BiLD: June 8th not possible (for Federico), will see
DIRAC+Rucio WS https://indico.cern.ch/event/1252369/ :
- Registration is open, do register, pay fee, take flights etc.
- several info added
- block booking of few hotels
- note for VISA: https://indico.cern.ch/event/1252369/page/29513-visa-information
LHCbDIRAC
- v11.0: deploy board in https://trello.com/b/Ep0PAkbv/deploy-110
- Singularity CE everywhere?
- Chris B “CTA apparently has something similar to the lbmcsubmit stuff so it might make sense to move some of my gitlab <-> dirac production system stuff to vanilla dirac.”
- https://lhcb-auth.web.cern.ch/
- The IAM instance was recreated (v1.8.1) to get proper robot user accounts (they were previously bound to a user account).
- Still one minor issue to be solved and then we will create 2 clients: one for the LHCbDIRAC certification, one for the ETF tests.
- from previous meeting Alexandre An admin can allow a user (a client actually) to get access to the
compute.cancel/create/modifyscopes. Chris allowed Andrei and I to get access to the compute scopes, thus we should be able to use our clients to submit pilots.
- Federico I suggest to have a LHCbDIRAC hackathon with the only real goal to verify the token pilot submission. This would verify that:
- what’s in the code is correct not only for wlcg
- the instructions are correct
- LHCb IaM instance is correctly configured
- LHCb has few resources that accept tokens
- In CC we should have one