BiLD-Dev

Thursday 25 Jan 2024, 10:00 → 12:00 Europe/Zurich

2/R-014 (CERN)

Federico Stagni (CERN)

Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.

Zoom: BiLD
https://cern.zoom.us/j/62504856418?pwd=TU1kb01SOFFpSDBJeWVBdU9qemVXQT09

Meeting ID: 62504856418
Passcode: 12345678
 

working notes

BiLD – 25/01/2024

At CERN: Federico, Andrè, Alexandre, Christophe, Alvaro
On Zoom: Christopher, Janusz, Daniela, Simon, Andrei, Ewoud, Hideki, Janusz, Xiaomei, Cedric
Apologies:

Follow-up from previous meetings

• Last BiLD on December 14th 2023
  • So, happy new year!
• Last DIRAC certification hackathon 1 week ago
  • Did not run fully, mostly concentrating on WMS
  • Clients were installed using a pre-version of DIRACOS2
  •  Daniela My DN nearly got suspended as 275134 pilot jobs were submitted to RAL-LCG2 and no-one caught it.
    •  Federico indeed, it shoud have been caught immediately!
    • Solution is being coded in https://github.com/DIRACGrid/DIRAC/pull/7414
      •  Alexandre Not fully understood
      •  Igor I have seen similar issues with our HPC – this uses SSHCE
        • this is a different issue
    • For the moment, SiteDirectors are stopped
• DIRAC+X hackathon Tue and Wed this week
  • Federico, Andrei, Christophe, Alexandre, Janusz, Cedric, Ruslan, Alvaro present
  • Worked in pairs, mostly, on different topics, DIRAC and DiracX

DIRAC communities roundtable

LHCb:

 Federico+Alexandre+Christophe+Christopher+Alvaro

• Running on latest v8 patch
• Trying to update LHCbDIRAC certification to v9 – this needs DiracX first, instructions will come soon
  • There, Pilot devel version will be tried out

Belle2

 Cedric, Hideki

• Production instance: v7.3
  • applying a workaround for HTCondor CEs w/o X509
• Pilot? Updated to use Vanilla master?
  • testing
  • also still using py2
  • some issues observed at DESY, under investigation

ILC/Calice/FCC

 Andre

• NTR

EGI

 Andrei

• NTR

Juno/BES3

 Xiaomei

• Running v8 in prod. No issues
• Started to use Tape – no experience
  • 2 sites using CTA
  •  Christophe please open a discussion on Github regarding your use case

JINR

 Igor

• Running pilot devel since a while. Few issues on the road, now seemingly working fine.

GridPP:

 Simon, Janusz, Daniela

• No updates to production server. Rolling out pilot jobs with tokens across the UK on production server.

Topics from GitHub/Discussions

only un-answered topics with discussion updates:

• NTR

DIRAC releases

• v8r0
  • v8.0.36
    • Just “fixes”
• v9r0
  • Several simplification being added to WMS (mostly content of JobDB)
  •  Federico simplified the update script (now 1 script instead of 4!).

DIRAC projects

DIRAC:

Issues by milestone:

• v8.0:

  • 10+ open issues
    • usual reminder for closing/moving old ones

• v9.0:

  • 15+ open issues
  • Those for Elastic Parameters DB are under scrutiny
    •  Federico I made some research, and I still do not see a better solution than Elastic for our use case
      •  Chris+Chris MySQL with JSON should be alright

Other issues:

•  Janusz TornadoPilotLogging is too resource intensive
  • This might have been hampered by the SiteDirector issues (250k+ pilots submitted)
  • Buffer is now adjustable
  • Added possibility to enable logging from a specific CE (list)
    • documentation should be added
  • Found some other issue when running github actions integration tests, VO had to be added (not completed yet)
• [8.0] Implementation of metadata methods into RucioFileCatalogClient closes issue #7382
• Synchronize IAM to CS from VOMS2CSAgent
  • does not seem difficult
• PUSP can be dropped
• from previous meeting https://github.com/DIRACGrid/DIRAC/issues/7260
  • Solved (but in pre-release? of DIRACOS)

PRs discussed:
*

WebApp:

•  Federico I created a few simple PRs for integration to adapt to WMS changes

Pilot:

• Sadly, no merge of devel to master yet. Will first test in LHCbDIRAC certification (today/tomorrow or next week) then LHCb production setup

DIRACOS:

• New pre-release last week.
• DIRACOS: latest versions are dropping the support for v7r3
  • Only Belle2 affected, running with a fixed old version of DIRACOS
  •  Hideki Still https://github.com/DIRACGrid/DIRAC/discussions/7369 and we have issues running v8 with latest DIRACOS

Documentation:

• Every once in a while issue popping up about circular imports

OAuth2:

• NTR

management

• 2 issues left, closed 2 since last time
• Always upload releases to CVMFS is again valid for clients
  • Should also be done for non-x86 versions.
    •  Andrei Working on this, might need to ask a few questions to Cristopher
  • Also the upload of pilot files to CVMFS could be done same way
  • Right now the cron-job runs hourly
  • Few issues discovered (the releases were not properly propagated – CVMFS issue at RAL)
    • from Stratum0 to Stratum1
    • fixed by hand from time to time
    • https://cvmfs-monitor-frontend.web.cern.ch/ might be needed

diraccfg

• NTR

DB12

 Alexandre

• from previous meeting
  •  Ewoud opened PR for py3.11 (in progress)
    • missing the Intel vs AMD
  •  Federico Can we have DB12 run on ARM
    • Andrè : I access two “real” ARM machines through OpenStack that can be used for the tests
    • Federico: passed to Ewoud
    • hepscore and ARM: https://ggus.eu/index.php?mode=ticket_info&ticket_id=164939 (Imperial’s ARM is still very much in the “setting it up stage”

Rucio

•  Cedric PR for the MetaData catalog

Tests

•  Federico I will create a new repo for running “nightly” system tests targeting the DIRAC cert setup

Release planning, tests and certification

• Certification machines

  • lbcertifdirac70 machine:
    • we should move to Alma9 box in Q1 2024 (CC7 EOL June 2024)
      • As said, this should be done outside of CERN
  • MySQL update: we need to schedule an update to most recent MySQL version
    • there does not seem to be a specific stopper

• Next hackathon(s)

  • in 2 weeks

AOB

Next BiLDX: Feb 1st
Next hackathon: Feb 8th
Next BiLD: Feb 15th

• Next DIRAC Users’ workshop in Lyon https://indico.cern.ch/event/1341205 June 19th

• Next DIRAC+X hackathon: will call one before the DIRAC workshop

---

LHCbDIRAC

• v11.0: deploy board in https://trello.com/b/Ep0PAkbv/deploy-110
  • NTR
• Bookkeping MRs:
  •  Federico will need some help for reviewing
• VOMS backend will be shutdown in June 24
  • we need to make sure that all links will be updated
  •  Christophe will do adiabatically over the coming weeks
• OpenSearch migration
  •  Chris can we drop some indices to ease the migration?
    • older logs can be removed?
•  Federico Alma9
  • CC7 EOL June 24
  • 2 machines (lbvobox900 and lbvobox901) created by Joel
  • https://github.com/DIRACGrid/DIRAC/issues/7368

10:00 → 10:10 Items from Previous BiLD-Dev

10:10 → 10:20 DIRAC Communities roundtable

10:20 → 10:30 DIRAC releases

10:30 → 10:55 DIRAC projects

• DIRAC
• WebApp
• Pilot
• DIRACOS2
• VMDIRAC
• Documentation
• OAuth2
• DiracX
• other externals (include Rucio)

DIRAC certification

• Certification setups
• Cert task board teamplate
• Slack channel

10:55 → 11:00 Release planning, tests and certification

DIRAC PRs

11:00 → 11:15 Weekly development(s) focus

11:15 → 11:25 AOB

Convener: Federico Stagni (CERN)

11:25 → 11:40 LHCbDIRAC