Dops – 12/02/2026
At CERN: Federico, Christophe, Alexandre
On Zoom: Janusz, Hideki, Xiaomei, Michel, Dhiraj, Ryun, Luisa, Hideki, Heloise, Stella, Loris, Andrei, Vladimir
Apologies: Daniela, Simon
! Welcome to the first Dops (Dirac’s “Operations”) meeting !
Ddev meetings are about SCRUM sprints (board here), and happen once a week
In Dops we discuss (every 4th week):
- communities issues and requests
- light sprint reviews through:
- releases announcements and reviews
- prioritizations of communities’ feature requests
- longer term developers’ issues that need agreement and/or inputs
- relations with Rucio and/or other projects or entities
- whatever happens in the Dirac(X) world
Connected to this:
- The group/ML “diracproject-users” have been expanded to include all known active contributors/developers/admins. This groups serves for general announcements
- Federico created a new ML “diracproject-admins” with (mostly) 1 admin per installation. Preferrably only 1 because:
- avoiding spamming people
- making sure that that one (the only contact) replies, without thinking the other one(s) might/should reply instead
- Few topics from today come from questions posted in the above ML
Previous meetings
Communities issues and requests : roundtable
LHCb:
Federico+Christopher+Christophe+Alexandre+Ryun
- Kept installing latest production release
- Regularly running 300K+ jobs, with peaks of 500K
Belle2
Hideki, Ueda
- Migrated our databases, both MySQL and OpenSearch to newer versions
- Pilot submission: will move to token authN
Juno+BES3:
Xiaomei
- Heavy productions since August. Running on 2 servers.
- “only 20k” running jobs, but high job frequency because of short jobs are putting pressure on SandboxStore and CS
- Federico + Christophe solutions in v8 are mostly outside of DIRAC itself:
- the performances of the disk are critical
- in LHCb we created a DNS load balancer
- increase the validity of the CS (option to increase the refresh time)
CTAO
Luisa
- This years we are going to deploy pre-production installation fully on kubernetes, in the CTAO data center
GridPP:
Daniela, Simon
- No changes to prod.
- Still owe everyone a container writeup from the workshop.
Releases announcements and reviews
DIRAC
DiracX
- v0.0.8
- dense but “technical” release
DiracOS
- 2.58
- FEAT: (#154) Use pixi for dependency management and build tasks
- NEW: (#152) fabric and related dependencies for the SSHCE
Dirac-CWL
- Loris prepared a presentation, attached to the agenda.
- CWL is there to ultimately replace XML and JDL formats
- exprimental prototype. CTAO and LHCb on it
- jobs execution (not submission) via CWL
Feature requests, and developers’ issues: inputs and prioritizations from communities
Jobs rescheduling
Do we need to reschedule jobs? Original issue: https://github.com/DIRACGrid/diracx/issues/760. Can we instead resubmit jobs?
- Federico I asked admins to give their opinion on this. Summaries of the answers I got:
- Daniela+Simon
- if re-submiting the failed job and if then this new job has a new job id, the end user has now lost track of their jobs (replfail & resubmit e.g. once is OK)
- It would be good to have the error of any failures available, even if it succeeds at the next attempt.
- Xiaomei
- don’t oppose the eventual removal of rescheduling, we need a clear timeline for its complete decommissioning. Its removal must be coordinated with sufficient time for us to properly tune and test new functionalities. It would also be extremely helpful to have concrete examples or guidelines for implementing changes within the transformation system.
- Luisa
- the large majority of rescheduling is due to the JobWrapper not being able to resolve InputData. From what I’ve observed, most of the jobs that can’t access InputData at the first trial, they don’t succeed even after
- Could:
- replace automatic rescheduling by automatic resubmission
- do not support automatic rescheduling but enable the possibility to manually resubmit failed (or selected) jobs, i.e. submit new jobs identical to the selected ones
- André
- Ueda
- Attached the following Slides
- rescheduling is important in several aspects
- there’s also “resetJob” action. Should that be removed too?
- Andrei
- proposal:
- Keep rescheduling as it is now but consider it an internal WMS fallback mechanism that can not be accessed by users. “Reset” only for power users.
- Think of a tool that would “reschedule” input files to generate new jobs.
The discussion will be summarized in the GH issue. Adjurned
RSS in DIRAC, for DiracX
Possible breaking changes in diracx
- https://github.com/DIRACGrid/diracx/issues/781 would introduce breaking changes. It would only affect already created diracx extensions. @all, would you be affected?
- –> basically, greeen light as no-one but LHCb runs atm diracx in production
Prioritized backlog: communities input
https://github.com/orgs/DIRACGrid/projects/30/views/3 contains the prioritized backlog.
AOB
Next appointments