Choose timezone
Your profile timezone:
BiLD-Dev
→
Europe/Zurich
Description
Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.
Join Zoom Meeting
https://cern.zoom.us/j/91083694183?pwd=ZkdDY1I1YkJVc2o3UTdBY1BRZE15UT09
Meeting ID: 910 8369 4183
Passcode: 12345678
One tap mobile
+41432107108,,91083694183# Switzerland
+41315280988,,91083694183# Switzerland
Dial by your location
+41 43 210 71 08 Switzerland
+41 31 528 09 88 Switzerland
+41 43 210 70 42 Switzerland
+33 1 8699 5831 France
+33 1 7037 2246 France
+33 1 7037 9729 France
Meeting ID: 910 8369 4183
Find your local number: https://cern.zoom.us/u/artUfAdNB
Join by SIP
91083694183@188.184.89.188
91083694183@188.185.118.153
Join by H.323
188.184.89.188
188.185.118.153
Meeting ID: 910 8369 4183
Passcode: 12345678
BiLD (Bi-weekly DIRAC Development meeting) – 03/11/2022
On Zoom: Federico, André, Andrei, Alexey, Alexandre, Christopher, Daniela, Hideki, Igor, Michel, Simon, Janusz
Apologies: Xiaomei
Follow-up from previous meetings
- hackathon October 27th, on 8.1.0a3:
- Some fixes found, committed/merged
- Last BiLD 2 weeks ago
- all updates below
DIRAC communities roundtable
LHCb:
Federico+Alexandre+Christopher+Alexey
- Running in production v7.3.32
- PR #6477 that backported ElasticJobParametersDB from rel-v8r0 branch was necessary because the single index used in LHCb was too large
ILC/CLIC/FCC/Calice:
André
- NTR
EGI:
Andrei
- NTR
Belle2
Hideki+Ueda
- Production on v7r2. Ready to use ElasticSearch
NICA
Igor
https://github.com/DIRACGrid/DIRAC/issues/6474 (see below)
during transfer to xrootd DIRAC uses parameter xrd.gsiproxy=/tmp/x509. In our JINR installation these transfers are blocked. Using parameter xrd.wantprot=gsi,unix seems to work stable. Issue opened in xrootd repository.
GridPP:
Daniela+Simon+Janusz
- No changes in production: v7.3.26 on production server
- v8.0.2 in testing: deployed with a matching indigo iam server for token testing
- (incidentally DIRAC UI seems to work fine if installed on Rocky 9)
Topics from GitHub/Discussions or Google forum
only un-answered topics below:
- Personal certificates with “sn” and “gn” fields not supported?
- Maybe update to VOMS2CSAgent?
- Pilots/ARC “proxy not found”
- Andrei we should simply put by the default the bundling of the proxy in the pilots
- How to add a new Subsection to an existing part of the Configuration
- needs input
- Expose rucio API via DIRAC (DMS/Client/RucioClient.py)
- unanswered. Input expected from Cedric.
DIRAC releases
- v7r3
- v7.3.32
- WorkloadManagementSystem
- NEW: (#6477) backported ElasticJobParametersDB from rel-v8r0 branch
- FIX: (#6469) reduce the PollingTime of JobAgent to 20s
- Core
- FIX: (#6468) quoting the DB passwords for special characters
- Resources
- NEW: Added AREXComputingElement
- WorkloadManagementSystem
- v7.3.32
- v8r0
- v8.0.3
- Will be created when current PRs are approved+merged
- v8.0.3
- v8r1
- v8.1.0a3
- CHANGE: (#6451) remove support for myProxy
- CHANGE: (#6380) Remove VMDIRAC components
- v8.1.0a3
DIRAC projects
DIRAC:
Issues by milestone:
- v7r3:
- Still 7 open issues (cut in half since last meeting)
- Add possibility to specify a DIRACOS2 version in SystemAdministrator and Pilot.
- often requested
- v8.0:
- 14 open issues, some of them will need to be re-assigned, and/or moved
- v8.1:
- 12 open issues
Other issues:
PRs discussed:
- [integration] Remote Pilot Logger to Tornado
- Janusz and Federico discussed plan of action:
- move to targeting 8.1 (done)
- reach a “merge-able point”, even if not “perfect”
- try it out in DIRAC certification setup
- https://github.com/DIRACGrid/DIRAC/issues/6470 for later
- need to identify the VO from a proxy: created https://github.com/DIRACGrid/DIRAC/issues/6491
- Janusz and Federico discussed plan of action:
WebApp:
- 4 PRs open, mostlty waiting for reviews
Pilot:
- feat: Enable remote pilot logging using HTTP interface to Tornado
- Will need to be picked up by GitHub Action tests.
DIRACOS2:
- 3 new releases in the past 2 weeks:
- 2.29
- Downgraded
arclibrary due to buggy arc release reported here - Update tests to test arc even more thoroughly
- Downgraded
- 2.28
- Fix issue generating macOS installer
- Add missing arc dependency
- Update tests to test arc more thoroughly
- 2.27
- NEW: (#81) Include apptainer as a singularity alternative.
- 2.29
- from previous meeting Add htcondor-utils current version (9.11.2), fix diracosrc to use GSI and SCITOKENS auth methods. PR will be added for the latter.
- There is no version of HTCondor that supports both
- Christopher we can ask for re-packaging of old versions in Conda.
- from previous meeting Christopher if anyone is interested in making a CVMFS-based environment, let me know I will give instructions
Documentation:
- Few minor fixes.
OAuth2:
- from previous meeting WLCG timeline document: https://zenodo.org/record/7014668#.YyLag9JBwQ9
tornado/HTTPs
- Created 3 new tornado handlers in PR 6450, for FrameworkSystem
- from previous meeting v8.0.1 WebApp ssl issue
- PR with fix (draft) created in WebApp, Chistopher care to test?
- Simon I tested and looks OK
- Christopher I would still not be very confident about merging this PR
- Christophe …did not we say in the past that NGINX is becoming mandatory?
- Daniela if we can get away without it, it would be better
- There’s no log about this decision even though it was definitely discussed.
- Christophe
tornadois crap for download, nginx “needed” in this sense
- Andrei do we make nginx mandatory? In this case we should provide proper instructions. Then it should be run on every host (also for https services)
- Christophe …did not we say in the past that NGINX is becoming mandatory?
- 03/11 Daniela As a side note: GridPP would probably run this with apache as we use that elsewhere.
- Federico in practice what does that imply?
- prob trivial.
- Federico in practice what does that imply?
- PR with fix (draft) created in WebApp, Chistopher care to test?
management
- from previous meeting 3 issues left, still valid
diraccfg
- from previous meeting Christopher Do we want to make a release that drops support for py2?
- yes
COMDIRAC
- Daniela Still working on merge (basically going through the commands one by one, fixing them up as we go along, hoping to evade the wrath of Federico™).
- For discussion:
dinput(command to retrieve input sandbox or jdl): Can we drop this ?- Andrei for the moment it can be dropped
DB12
- NTR
Rucio
- NTR
Release planning, tests and certification
8.1 integration tests:
- feat: removed DIRAC_USE_JSON_DECODE, flipped DIRAC_USE_JSON_ENCODE
- HTTPs as default (plus need to be revisited how it is done)
Certification machines
- lbcertifdirac70 machine:
- Computing Element of each supported type should be configured (HTCondorCE, 2 ARCS (Alexandre (maybe use RALPP and Glasgow ? I can talk to them if there are issues --Daniela)), 1 Cloud (Daniela - this is already part of the tests), SSH). Consider OAuth tests as well - what is needed as far as certification servers configuration is concerned.
- lbcertifdiracoauth machine:
*
Next hackathon(s)
- Next week, on lbcertifidirac70 and with v8.1.a0X
AOB
- CHEP2023 seems to be happening for real this time. Abstract submission deadline November 17th (very soon!). Leaving it here, proposals welcome.
- prob extended to November 28th
Next hackathon: November 10th
Next BiLD: November 17th
LHCbDIRAC
- v10r4: deploy board in https://trello.com/b/kzUKdMts/deploy-v10r3
lbprodupdate: should we move ProxyDB to another instance (separate from the rest)?
- https://lhcb-auth.web.cern.ch/
- We need to have IaM as interface of VOMS (some code need to be developed here)
- For the actual operation we should see (up to Joel).
- Andrei this is not properly configured yet. Not usable for Pilot submission yet (compute scopes not added).
- Christophe is involved, probably should wait for him.
- LHCbDIRAC hackathon based on 8.0
- Next when? – next Tuesday at 14:00
- At some point (after it will be done on DIRAC certification) we need to setup a CE that accepts token and try out our IAM instance.
- ISIMA projects anyone?
- not this year
There are minutes attached to this event.
Show them.