BiLD (Bi-weekly DIRAC Development meeting) – 17/09/2020
At CERN: A few of us, but each in our office ;-)
On Zoom: Federico, Alexandre, André, Andrei, Andrii, Cedric, Christophe, Christopher, Dan Whitehouse (GridPP), Daniela, Marko, Simon, Vladimir, Janusz, Xiaomei
Apologies: Igor
Follow-up from previous meeting
After OAuth pres (2 BiLD meetings ago - 7 weeks ago):
- REST interface: RFC for discussion? (see today)
- Basic OAuth library:
- Chris Burr started drafting something. “More or less I have a proposal, hope to open an issue before holidays”
- JWT: suggestion in https://github.com/DIRACGrid/DIRAC/issues/4741 (Session authentication for REST endpoints)
- RFC for semantic of components?
- not started yet [Andrei/Federico/Andrii]
After HTTps presentation (1 BiLD meeting ago - 5 weeks ago)
A few updates, but the PR looks ready to be merged.
management devs
DIRAC communities roundtable
GridPP:
- Nothing new.
- Was waiting for the singularity fixes (merged in in v7r1p12) before putting v7r1 in production.
CLIC:
- Still on v6r22. Waiting for last moves out of LFC
- Also have to move from Castor to CTA, in contact with Chris and CERN IT, will wait for LHCb to move
LHCb:
- Still on v7r0. We’ll restart hackathons for v7r1 and move soon (prob in 3 weeks in production).
- Castor to CTA: should not need any DIRAC change
- Except master CS and WebApp we are running with M2Crypto everywhere. Performances to be seen in the long run.
France Grilles:
- Running v7r0
- Started looking at ElasticSearch
- Federico: which version? DIRAC can interact with ES6, not ES7.
EGI:
- Still struggling with AuthN, CheckIN in conjuction with other community management tools (perun)
- FileCatalog management for one VO in progress
Belle2:
Nica:
Juno, BES3:
- MC transformations running, using MCExtensionAgent for automated extension
Current situation
DIRAC
- v6r22:
- v7r0:
- v7r1:
- v7r1p12
- Fixes for JSON serialization
- SingularityCE:
- Use singularity userns flag if possible and allow CVMFS fallback
- added possibility to run without re-installing DIRAC inside the container
- NEW: RSS DB schema change to include a VO column (https://github.com/DIRACGrid/DIRAC/pull/4720)
- Need to add it to the release wiki
- Andrei: we need to do it “by hand” right now, we can code a simple tool for doing it
- Chris: we can use external tools for that
- Andrei: for very simple cases we don’t external tools
- v7r2:
- v7r2-pre14 created, just including the above
WebApp:
Pilot3:
- One draft PR open for tests of various types of CEs, was waiting for v7r1p12 for starting to use it.
DIRACOS:
VMDIRAC:
Documentation:
OAuth2:
- The PR 4650 will be rebased on top of Tornado PR.
- The PR will then target v7r3
tornado
-
Tornado PR: ready to be integrated, just one last review needed [Federico]
-
JSON serialization: only 2 tasks left open, rest is in v7r1
-
We should move tornado fork in DIRACGRid repo [Christophe]
other externals
- M2Crypto:
- still one open PR on M2Crypto side – but anyway it seems like we don’t need it
management
- We should to move the containers to new GitHub container registry [Marko]
diraccfg
- One PR for exposing version parsing (already merged)
Release planning, tests and certification
Federico proposed:
- Go back to v7r1 for certifying:
- MySQL 8 (the DIRAC certification server was updated already, this should not be an issue)
- The following flags (added in
extra_bashrc
)
export DIRAC_DEBUG_DENCODE_CALLSTACK=1
export DIRAC_USE_JSON_DECODE=Yes
- SingularityCE (~easy to test)
- JobParameters on ES (also controlled by a flag, right now turned on)
- ElasticSearch 7 (dev to be done, probably)
- Move to MultiVO installation, basically what is in https://github.com/DIRACGrid/DIRAC/issues/4631 (this needs work - GridPP?)
Points from 1 to 4 can be tested at the next hackathon that will be next week. Point 5 needs some work. Once done we can do point 6.
- v7r2:
- Include HTTPs
- Include Python 3 PRs (only the most recent one, as the others are included)
Start certification with the above 2 once v7r1 is done
Weekly development(s) focus
- Vladimir: we should suppress Glue1 from Bdii2CSAgent, because ARC CEs are just buggy, and they publish wrong info to Glue1 (see note in https://codimd.web.cern.ch/6r1SApCfTu2DlPYiZGuu_g#)
- Proposal: v7r1 send notification that it Glue1 will be removed, then later really remove it [Federico]
- Daniela: We have our extension, we pick things from Glue1 and Glue2
- Andre’: We use Glue2 only for HtCondorCEs
DIRAC: current PRs and tasks being worked on, or topics from Google forum
PRs:
- v7r0:
- v7r1:
- v7r2:
- Out of the 4 python3 PRs, only the most recent one will stay (will be rebased after the HTTPs PR)
On issues:
Topics from Google forum:
AOB
CERN policies still forbid travel. Can’t know yet when there will be next DUW.
EGI workshop in November will go fully virtual. 1 day DIRAC session? probably - AT in contact.
Next BiLD in 3 weeks.
LHCbDIRAC
- ES: before moving to ES7 we need to reduce the number or indices. Federico looking into it, some issues found. Still looking for the best way to do it.
- Hackathon just after the LHCb computing workshop