BiLD-Dev
Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.
Join Zoom Meeting
https://cern.zoom.us/j/91083694183?pwd=ZkdDY1I1YkJVc2o3UTdBY1BRZE15UT09
Meeting ID: 910 8369 4183
Passcode: 12345678
One tap mobile
+41432107108,,91083694183# Switzerland
+41315280988,,91083694183# Switzerland
Dial by your location
+41 43 210 71 08 Switzerland
+41 31 528 09 88 Switzerland
+41 43 210 70 42 Switzerland
+33 1 8699 5831 France
+33 1 7037 2246 France
+33 1 7037 9729 France
Meeting ID: 910 8369 4183
Find your local number: https://cern.zoom.us/u/artUfAdNB
Join by SIP
91083694183@188.184.89.188
91083694183@188.185.118.153
Join by H.323
188.184.89.188
188.185.118.153
Meeting ID: 910 8369 4183
Passcode: 12345678
BiLD (Bi-weekly DIRAC Development meeting) – 19/08/2021
At CERN: Christopher, André (in our offices…)
On Zoom: Federico, Andrei, Andrii, Cedric, Daniela, Hideki, Janusz, Patrick (CTA), Simon, Xiaomei
Apologies:
Follow-up from previous meetings
- Tried to run a hackathon last week:
- py3 hackathon on lbcertifdirac70 box
- messed up with the server installation, didn’t go much far
- Fixed yesterday in https://github.com/DIRACGrid/DIRAC/pull/5340
DIRAC communities roundtable
GridPP:
Daniela
- NTR
ILC/CLIC/FCC:
André
- Updating iLCDirac to python3 compatibility
LHCb:
Federico+Christophe+Christopher+Alexandre
- The default UI is py3 (on CVMFS)
- pilots still using Py2, by default
- 5% of them using python3 (including Singularity)
- pilots still using Py2, by default
- Had to remove several Sandboxes (incident described in PR)
EGI:
Andrei
- NTR
Belle2:
Hideki, Cedric
- Started migration test to v7r1
dirac-install.pyaccess CVMFS to retrieveglobalConfig.cfgbut we use our own. Maybe modify https://github.com/DIRACGrid/management/blob/master/globalDefaults.cfg#L26 would be better (this goes to CVMFS)
BES
Xiaomei
- Upgrading to v7r1, and moving from Pilot2 to Pilot3
- issue with Singularity, described in https://groups.google.com/g/diracgrid-forum/c/W-W569bQ9GU
- using GitLab inside IHEP (can’t use GitHub well from China)
- difficult to have fallback
- pilot files are anyway also put in CVMFS (/cvmfs/dirac.egi.eu/pilot) (cron)
CTA
Patrick
- upgraded to v7r2 some time ago.
DIRAC releases
- v7r2p20 + v7r2p21:
- Support Python 3 pilots in SingularityCE, use CernVM4 by default
- Fix unassigning sandboxes when removing jobs from the transformation system
- v7r3-pre20 + v7r3-pre21:
- removed src/DIRAC/Core/scripts/dirac-install.py
- Removed GlobusComputingElement
- Add a PilotStatus module to clearly define the pilot status in the code
DIRAC projects
DIRAC:
Issues by milestone:
- v7r1:
- no news/changes
- v7r2:
- v7r3:
- Pilot Status: Difference between Scheduled and Waiting? closed.
- Where should etc/ be for Python 3 based installations?
can be considered solved
Other issues:
- Moved 3 issues to new v8.0 milestone
PRs discussed:
- The old [v7r2] reuse Operations class for other section PR has been closed
- RFC prepared: https://github.com/DIRACGrid/DIRAC/wiki/Generalization-of-VO-and-Setup-specific-configuration-settings
- Discussed in https://github.com/DIRACGrid/DIRAC/issues/5287
- New draft PR created
- use
gConfig(create a new instance and a new dict depending on the VO, if specified)
- use
- Andrii If we agree with these changes, we need then to provide documentation on how to migrate the CS section structures
- André if the changes are only useful for WebApp, then it’s not worthy, let’s not do changes that are not needed
- (discussion went on for a bit)
- RFC prepared: https://github.com/DIRACGrid/DIRAC/wiki/Generalization-of-VO-and-Setup-specific-configuration-settings
WebApp:
- New releases, mostly fixes. Some issues can also be closed.
Pilot:
- PR(for
devel) for Remove singularity options
DIRACOS:
- Created new version (v1r26) for fixing the
elasticsearchclient (for supporting OpenSearch: issue described in https://aws.amazon.com/blogs/opensource/keeping-clients-of-opensearch-and-elasticsearch-compatible-with-open-source/)- at some point we’ll use the forked open clients, when they will be available (should come at https://github.com/orgs/opensearch-project/repositories)
DIRACOS2:
- from previous meeting
fts-reston py3?- NTR
- Created new version for fixing the
elasticsearchclient (for supporting OpenSearch)
Documentation:
- New theme on
integrationis there. It has ads.
OAuth2:
- from previous meeting Any news from issue WLCG and Token transition – reminders and requirements ?
- Still in progress, ongoing discussions on how Rucio uses tokens.
- from previous meeting Andrii forwarded https://docs.google.com/document/d/1ZQzElD866yV6t9AomeW6r-L9s-64pNB95OFDd1EZEbA/edit?ts=60814a09#heading=h.ur0csvxkk8tc which contains technical drawings about the proposed implementation.
- PR is always kept up-to-date
puppetPR is there
tornado/HTTPs
- NTR
management
- from previous meeting deploying on CVMFS
- Now should be done also for DIRACOS2
diraccfg
- NTR
COMDIRAC
- NTR
DB12
- The existing PR is technically ready for being merged, but py3 gives slightly different results that have not been properly investigated yet.
- Chris you should be surprised, as just the compilation of python can lead to different results.
- Federico still, we should make sure not to get big differences
- Chris you should be surprised, as just the compilation of python can lead to different results.
other externals, including Rucio
Release planning, tests and certification
-
Certification machines
- lbcertifdirac70 machine:
- CERN is moving from elasticsearch to opensearch. A new cluster has been created (https://es-dirac-certification.cern.ch/) and all users in diracproject-users are admin.
- This machine is staying as the machine from which production releases are derived. The security will keep being based on X509, and VOMS proxies. All hackathons on these machines will be following the “usual” path.
- lbcertifdiracoauth machine:
- The machine will be used for testing all interactions in the “tokens world”
- lbcertifdirac70 machine:
-
Next hackathon(s)
- We need to accelerate with testing of developments for tokens. The suggestion is to run the following 3 hackathons, in this order:
- pre: Move lbcertifdirac70 to py3 server. “Standardize” how we make py3 servers, and apply the same recipe for installing lbcertifdirac70 and lbcertifdiracoath
- this should be done (TBC tomorrow)
- on lbcertifdirac70: standard hackathon, on the v7r3-preX tag -> tomorrow! at 10am
- on lbcertifdirac70: standard hackathon, on the content of token PR -> in one week
- on lbcertifdiracoath: auth hackathon, on the content of token PR -> in 3 weeks
- pre: Move lbcertifdirac70 to py3 server. “Standardize” how we make py3 servers, and apply the same recipe for installing lbcertifdirac70 and lbcertifdiracoath
- We need to accelerate with testing of developments for tokens. The suggestion is to run the following 3 hackathons, in this order:
AOB and topics from Google forum
- GitHub is introducing “Discussions”: https://github.com/DIRACGrid/DIRAC/discussions/landing. Maybe replace the google forum?
- Daniela to participate in discussion, you would need an account on github, while for the google forum just a mail
- TBD at the next meeting
Next hackathon: tomorrow, 10 AM
Next BiLD on September 16th: https://indico.cern.ch/event/1065229/
LHCbDIRAC
- from previous meeting https://trello.com/b/BQvEbteg/deploy-v10r2
- ES stuff: we will migrate to OpenSearch, starting from the DIRAC certification, then LHCb certification, and only then Production
- for Run3 productions, the use of
ProdConfwill be most probably deprecated. Options will be in a JSON. Also, most probably we won’t uselb-runRunApplication.pyshould continue holding the logic. There is also a test for it that runs regularly in Jenkins, as part of the Pilot Workflow tests.