BiLD-Dev

Thursday 4 Apr 2024, 10:00 → 12:00 Europe/Zurich

2/R-014 (CERN)

Federico Stagni (CERN)

Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.

Zoom: BiLD
https://cern.zoom.us/j/62504856418?pwd=TU1kb01SOFFpSDBJeWVBdU9qemVXQT09

Meeting ID: 62504856418
Passcode: 12345678
 

working notes

BiLD – 04/04/2024

At CERN: Federico, André, Alexandre, Christophe
On Zoom: Janusz, Daniela (late as still mentally on holiday), Simon, Andrei, Igor, Hideki, Janusz, Vladimir, Alexey
Apologies:

Follow-up from previous meetings

• Last BiLD on March 7th
• Last DIRAC certification hackathon was supposed to happen March 21st, but was cancelled because of issues discovered before deploying the release
  •  Federico I promised to create a new release and install it, but I did not get a review of the needed PRs…
    • and, we did not “test” the board from GitHub
• Last BiLDx on March 14th
  • For now on, we are not planning regular BildX meetings anymore, but getting updates in this same meeting

DIRAC communities roundtable

LHCb:

 Federico+Alexandre+Christophe+Christopher

• Moving to EL9 boxes: started converting existing boxes.
• Moved (before Easter) the default Singularity container to EL9
  • we discovered a few sites that provided 15+ years old hardware:

  Red Hat Enterprise Linux 9 is distributed with the kernel version 5.14, which provides support for the following architectures at the minimum required version:

  • AMD and Intel 64-bit architectures (x86-64-v2)

  • and jobs were rescheduled because only x86-64 was there
  • we decided to do nothing and accept that we can not run there.
• ARM farm at Glasgow:
  • set it up, not yet ran production jobs

Belle2

 Hideki

• Still on v7.3, migration will be completed soon, also extension almost ready

ILC/Calice/FCC

 André

• NTR

EGI

 Andrei

• WMS system : still investigating, but from time to time there’s a spike of open files.

JINR

 Igor

• CTA usage: discussion on GitHub to be done. Main question is how to read the data
• Tested OpenNebula cloud – working smoothly
• Pilots are taking releases from CVMFS
  • loads on WNs drastically reduced

GridPP:

 Simon, Janusz, Daniela

• Nothing to report

Topics from GitHub/Discussions

only un-answered topics with discussion updates:

• Old ARC CE interface?
  • the configuration would need to move from “ARC” to “AREX” and in theory it would still work everywhere

DIRAC releases

• v8r0
  • NTR
• v9r0
  • NTR

DIRAC projects

DIRAC:

Issues by milestone:

• v8.0:
  • 10+ open issues, usual reminder for closing/moving old ones
  • No updates on existing ones recently
• v9.0:
  • 15+ open issues, as usual some movements

Other issues:

• [8.0] Implementation of metadata methods into RucioFileCatalogClient closes issue #7382
• CentOS 7 EOL
  • Almost completed

PRs discussed:

• [9.0] feat: introduce the JobWrapperLightTemplate for resources with no external connectivity
  • Only for MareNostrum. Controlled by SiteDirector and PushJobAgent
  •  André please change name with something more meaningful

DiracX:

• Progress on the Security policy (including the JobPolicy)
• Started preparing some tasks for the hackathon next week
• diracx-web:
  • new developer
  • 2 new PRs, already merged
    • JobMonitor: do operations on jobs
    • Integration tests (e.g. login and logout)
      • cypress framework (specific to React world)

WebApp:

• Just one PR: https://github.com/DIRACGrid/WebAppDIRAC/pull/756 reflecting changes in DIRAC for ProxyDB

Pilot:

• Merge of devel to master: done 12th March: https://github.com/DIRACGrid/DIRAC/discussions/7512
  •  Federico Did not receive any updates nor complains, so I am assuming everything went smoothly – or not?
    • VOs picking releases from CVMFS?
      • LHCb (from lhcb CVMFS repo)
      • ?
• New PR for devel completely and finally removing py2
• from previous meeting  Janusz some doc to write

DIRACOS:

• from previous meeting Made a new release with libxml2 downgraded. Issue opened to gfal2 for proper fix
  • FTS/gfal developers are going to look at this now
  • ==> a new release is there, pending in conda-forge
• Requests for Apple-M2 compatibility

Documentation:

• Building sometimes failing in PRs (not consistently)

OAuth2:

• NTR

management

• from previous meeting Always upload releases to CVMFS
  • news?
•  Federico I created el9 DIRAC container images with https://github.com/DIRACGrid/management/pull/61
  • then started using it as default image in https://github.com/DIRACGrid/DIRAC/pull/7513
• fix: use docker login should fix security issues, review requested

diraccfg

• NTR

DB12

 Alexandre

• from previous meeting
  •  Ewoud opened PR for py3.11 (in progress)
    • missing the Intel vs AMD
  •  Federico Can we have DB12 run on ARM
    • André : I access two “real” ARM machines through OpenStack that can be used for the tests
    • Federico: passed to Ewoud
    • hepscore and ARM: https://ggus.eu/index.php?mode=ticket_info&ticket_id=164939 (Imperial’s ARM is still very much in the “setting it up stage”

Rucio

• NTR

Tests

Added PRs for workflow and “submitAndMatch” tests, the main question would be “who can run these tests (they require cert and key secrets)”:

• [9.0] test: add submit and match test
• [8.0] Workflow tests
  Comments summary:
• ideally, we could run everything in a self-consistent way (this requires some work)
• for the moment, they could be run only by the few developers interested

Release planning, tests and certification

• from previous meeting Trello: up to 10 collaborators:

  •  Christopher proposes to use Github projects:
    • https://github.com/orgs/DIRACGrid/projects/9/
    • GitHub has a concept of templates so we could use that. Or we could use the script I used to import it to generate a new board for each hackathon from a YAML spec: https://gist.github.com/chrisburr/0c4f48421e02e01286696453710c3028
  • April 4th Looks appropriate, but no possibility of adding comments to the “cards” without transforming them into issues

•  Federico I added the ARM CE+queue to Glasgow and submitted a few “helloworld” jobs – all fine

• Certification machines

  • Vanessa created a Alma9 machine, accessible via gsissh and gridmapfile
    •  Federico asked if it would be possible to avoid gsissh and gridmapfile (adduse + ssh + sudoers should be enough…?)
    • We did not discuss about new DBs and so on

• Next hackathon(s)

  • using github “projects”

AOB

• Next DIRAC+X hackathon: https://indico.cern.ch/event/1376672/
  • next week, 9-10 April
  • Andrei, Bertrand, Luisa, Natthan all present

Next BiLD: in 3 weeks
Next hackathon: in 2 weeks?

• Next DIRAC Users’ workshop in Lyon https://indico.cern.ch/e/duw10 June 19th->21st
  • registrations open

---

LHCbDIRAC

• v11.0: deploy board in https://trello.com/b/Ep0PAkbv/deploy-110
  • NNTR
• Added the same workflow tests also added for DIRAC
• VOMS backend will be shutdown in June 24
  • from previous meeting we need to make sure that all links will be updated
    •  Christophe will do adiabatically over the coming weeks
• OpenSearch migration?
  • NNTR
• BKK MRs: should be ~ready

10:00 → 10:10 Items from Previous BiLD-Dev

10:10 → 10:20 DIRAC Communities roundtable

10:20 → 10:30 DIRAC releases

10:30 → 10:55 DIRAC projects

• DIRAC
• WebApp
• Pilot
• DIRACOS2
• VMDIRAC
• Documentation
• OAuth2
• DiracX
• other externals (include Rucio)

DIRAC certification

• Certification setups
• Cert task board teamplate
• Slack channel

10:55 → 11:00 Release planning, tests and certification

DIRAC PRs

11:00 → 11:15 Weekly development(s) focus

11:15 → 11:25 AOB

Convener: Federico Stagni (CERN)

11:25 → 11:40 LHCbDIRAC