BiLD-Dev
Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.
Join Zoom Meeting
https://cern.zoom.us/j/91083694183?pwd=ZkdDY1I1YkJVc2o3UTdBY1BRZE15UT09
Meeting ID: 910 8369 4183
Passcode: 12345678
One tap mobile
+41432107108,,91083694183# Switzerland
+41315280988,,91083694183# Switzerland
Dial by your location
+41 43 210 71 08 Switzerland
+41 31 528 09 88 Switzerland
+41 43 210 70 42 Switzerland
+33 1 8699 5831 France
+33 1 7037 2246 France
+33 1 7037 9729 France
Meeting ID: 910 8369 4183
Find your local number: https://cern.zoom.us/u/artUfAdNB
Join by SIP
91083694183@188.184.89.188
91083694183@188.185.118.153
Join by H.323
188.184.89.188
188.185.118.153
Meeting ID: 910 8369 4183
Passcode: 12345678
BiLD (Bi-weekly DIRAC Development meeting) – 22/07/2021
At CERN: Chistophe
On Zoom: Federico, Alexandre, André, Andrei, Andrii, Daniela, Imane, Igor, Janusz, Simon
Apologies:
Follow-up from previous meetings
- Ran hackathon last week:
- trello board here
- puppet profile fixed (py2 + py3 – bashrc content, mostly)
- should be added to documentation for those not using puppet
- Few issues with Monitoring + Accounting
- Issue with v7r2 and Sandbox fixed in https://github.com/DIRACGrid/DIRAC/pull/5270
- v7r2p14 has been retired because of this issue
- python3 pilots and Singularity are not working together (https://trello.com/c/69moOFTb)
DIRAC communities roundtable
GridPP:
Daniela
- No changes to production server
- Dev: testing on top of v7r2p15
- HTCondor MP: looks OK, but not conclusive tests.
- If you want us to test different types of clouds, please let us know and we’ll try.
ILC/CLIC/FCC:
André
- Working on upgrading to v7r2
LHCb:
Federico+Christophe+Alexandre
- NTR
EGI:
Andrei
- Moving to v7r2 (2 servers already moved – for ES)
- Using Pilot3
- Using GPU equipped nodes through SSHBatch CE
- A PR for fixing is ready (already hotfixed)
- Not using yet any specific JDL requirement
- The jobs definition isn’t anyway fully ready for GPU reqs
JINR:
Igor
- Some issues while running on Clouds
- hypothesys of too many proxy delegations, not confirmed
- Developed extension of JobLaunchpad for the needs of a single VO
DIRAC releases
- v7r2p16:
- NEW: (#5268) add currentStatus parameter to TransformationClient.setTransformationParameter
- v7r3-preX:
- Nothing major
DIRAC projects
DIRAC:
Issues by milestone:
- v7r1:
- Handling job status when pilot got killed solved but in v7r2 (PR not merged yet)
- still some old documentation request
- #5203 can be closed?
- PFN is still used
- (Payload proxy renewal mechanism is screwed when using M2Crypto)[]
- Christophe will do this week
- v7r2:
- v7r3:
- Progress for Command line script changes
- Progress for tornado-based https JobStateUpdate handler
- new issue: “memory leak” from gLogger
- Christophe I am looking for a solution, maybe today
Other issues:
- 2 questions for HTCondorCE and not only: (#5277 + #5278)
- DIRAC’s Epoch time has time zone dependence
- it’s ugly but maybe we should just bite the bullet, in a major release
PRs discussed:
- [v7r2] reuse Operations class for other section
- specific comment
- RFC prepared: https://github.com/DIRACGrid/DIRAC/wiki/approach-to-configuration-navigation-depending-on-VO-and-setups
- Andrei I would even go to the very end and making
gConfigworks likeOperations(but this would be for v7r3) - the PR as it is keeps strict backward compatibility
- [v7r3] Merge VMDIRAC project
- Federico IMHO this is ready for merge. Notes in v7r3 wiki should be added. I would derive just a few tasks, for later:
- CloudDirector: can it be more similar to SiteDirector?
- Integration tests are completely missing, to add
- Create a VMs StateMachine
- Federico IMHO this is ready for merge. Notes in v7r3 wiki should be added. I would derive just a few tasks, for later:
- [v7r3] move Script goods to DIRACScript
- Federico IMHO this is ready for merge
- ON HOLD PRs merging strategy
WebApp:
- NTR
- Would need to sort out about adding VMDIRAC WebHandler to WebApp (https://github.com/DIRACGrid/WebAppDIRAC/issues/478)
Pilot:
- PR: Added parser of DIRAC version for py2/3 compatibility targeting
devel. Please review.
DIRACOS:
xroot5?- NTR
condor 9?- NTR
DIRACOS2:
- from previous meeting
fts-reston py3?- NTR
- from previous meeting Andrii jwt + other token stuff need to be added here
- NTR
- New release 2.8
- CI was not doing what was expected, fixed
- Andrei Installer also available from CVMFS, we should check the code for that (this is done with a simple cron)
- Andrei Made a py3 CVMFS client installation (in EGI CVMFS)
VMDIRAC:
- discussed above
Documentation:
- André Some issue in v7r1 for
rucionot being there
OAuth2:
- from previous meeting Any news from issue WLCG and Token transition – reminders and requirements ?
- meeting today about storage access
- from previous meeting Andrii forwarded https://docs.google.com/document/d/1ZQzElD866yV6t9AomeW6r-L9s-64pNB95OFDd1EZEbA/edit?ts=60814a09#heading=h.ur0csvxkk8tc which contains technical drawings about the proposed implementation.
tornado/HTTPs
- Running tornado HTTPS services on py3: (PR)
- Added 3 WMS HTTPs services in PR for v7r3 (depends from the above)
management
- from previous meeting deploying on CVMFS
- Now should be done also for DIRACOS2
diraccfg
- NTR
COMDIRAC
- NTR
DB12
- report from Imane (presentation on the agenda)
- 2 parts: development + analysis
- Development:
- DB12 ported to py2+3, added unit and integration tests
- published on pypi
- should be used in DIRAC
- no changes in the logic
- Analysis (to do):
- collect large amount data and verify if something should be changed.
- Adapt to MultiCore
- Development:
- 2 parts: development + analysis
- Igor We ran an analysis of the jobs submitted by a single user, collecting data from JobDB and JobParameters. It looks like AMD processors are underestimated by DB12
- Will share the DB query
other externals, including Rucio
- from previous meeting Issue: [URGENT] Rucio MUST be an optional dependency (#5250)
- NNTR
Release planning, tests and certification
-
Certification machines
- lbcertifdirac70 machine:
- py2 installation, for now
- lbcertifdiracoath machine:
- Andrii seems to be ready. One issue on how to organize Registry.
- cient for IAM (approved by Andrea), VO WLCG (not a admin yet) and one client for Check-IN.
- lbcertifdirac70 machine:
-
Next hackathon(s)
- Federico Supposing we merge the following PRs, I suggest to do one more (last) on py2 server:
- VMDIRAC
- #5035 (Script -> DIRACScript)
- the 2 HTTPs PRs
- Federico Supposing we merge the following PRs, I suggest to do one more (last) on py2 server:
AOB and topics from Google forum
Next hackathon on July 29th:
Next BiLD on August 05th:
LHCbDIRAC
- https://trello.com/b/BQvEbteg/deploy-v10r2
- No changes in the last 2 weeks
- from previous meeting python3 client on CVMFS
- Ganga and PID-calib using it
- Christophe keep bumping in division errors
- after release we can make it the default, also for the pilot
- from previous meetingSandboxStore: installed in prod
- xx% migrated -
rsync-ing - disabled the cleaning
- xx% migrated -
- we should start LHCbDIRAC hackathon based on v7r3/py2 and later py3