Dops + Ddev

Thursday 12 Feb 2026, 10:00 → 12:00 Europe/Zurich

2/R-014 (CERN)

The monthly Dops meeting (Dirac(X) operations) will run just before the weekly Ddev (Dirac(X) developers) meeting. 

Dops cwl presentation.pdf

Dops – 12/02/2026

At CERN: Federico, Christophe, Alexandre
On Zoom: Janusz, Hideki, Xiaomei, Michel, Dhiraj, Ryun, Luisa, Hideki, Heloise, Stella, Loris, Andrei, Vladimir
Apologies: Daniela, Simon

---

! Welcome to the first Dops (Dirac’s “Operations”) meeting !

Ddev meetings are about SCRUM sprints (board here), and happen once a week

Sprint

 

 

 

 

 

Product Backlog

Planning

Execution

Review

Retrospective

In Dops we discuss (every 4th week):

• communities issues and requests
• light sprint reviews through:
  • releases announcements and reviews
  • prioritizations of communities’ feature requests
• longer term developers’ issues that need agreement and/or inputs
• relations with Rucio and/or other projects or entities
• whatever happens in the Dirac(X) world

Connected to this:

• The group/ML “diracproject-users” have been expanded to include all known active contributors/developers/admins. This groups serves for general announcements
•  Federico created a new ML “diracproject-admins” with (mostly) 1 admin per installation. Preferrably only 1 because:
  • avoiding spamming people
  • making sure that that one (the only contact) replies, without thinking the other one(s) might/should reply instead
• Few topics from today come from questions posted in the above ML

---

Previous meetings

• We had a hackathon few weeks ago: https://indico.cern.ch/event/1582395/
  • 18(!) people in the room, +2 on zoom for the initial presentation
• Regular Ddev meetings are being held.

---

Communities issues and requests : roundtable

LHCb:

 Federico+Christopher+Christophe+Alexandre+Ryun

• Kept installing latest production release
• Regularly running 300K+ jobs, with peaks of 500K
  • WMS/Matcher service rather stressed
    • reminder: can not duplicate this one
  • Framework/ProxyManager services also under stress
    • Many instances running
  • Our WMSHistory plots are terrible, needs https://github.com/DIRACGrid/DIRAC/pull/8199

Belle2

 Hideki, Ueda

• Migrated our databases, both MySQL and OpenSearch to newer versions
• Pilot submission: will move to token authN

Juno+BES3:

 Xiaomei

• Heavy productions since August. Running on 2 servers.
  • “only 20k” running jobs, but high job frequency because of short jobs are putting pressure on SandboxStore and CS
    •  Federico + Christophe solutions in v8 are mostly outside of DIRAC itself:
      • the performances of the disk are critical
      • in LHCb we created a DNS load balancer
    • increase the validity of the CS (option to increase the refresh time)

CTAO

 Luisa

• This years we are going to deploy pre-production installation fully on kubernetes, in the CTAO data center

GridPP:

 Daniela, Simon

• No changes to prod.
• Still owe everyone a container writeup from the workshop.

---

Releases announcements and reviews

DIRAC

• v8.0.78

  • Core
    • CHANGE: (#8432) Support Python 3.14
  • WorkloadManagement
    • FIX: (#8427) JobAgent - do not fail already rescheduled job

• v9.0.15 (should have been called v9.1.0!)

  • WorkloadManagementSystem
    • NEW: (#8414) added a new table to TaskQueueDB for RAM requirements and matching
      • Do report possible issues (which might come e.g. from MySQL permissions)
    • NEW: (#8366) InProcessComputingElement takes RAM requirements into consideration for jobs’ matching
    • NEW: (#8366) PoolComputingElement takes RAM requirements into consideration when subdividing the pool
    • NEW: (#8366) SingularityComputingElement can enforce CG2 RAM limits
    • NEW: (#8399) ability to adjust job wrapper log level
  • Framework
    • CHANGE: (#8403) (SystemAdministrator): add the possibility to install non-released software versions
  • Interfaces
    • CHANGE: (#8407) Dirac().getInputDataCatalog - allow changing the inputDataPolicy
  • TS
    • CHANGE: (#8404) TransformationAgent does not bulk calls to dm.getReplicas
  • DMS + RMS
    • CHANGE: (#8402) batch getReplicas at the FileCatalogClient level instead of at the DM level for speedup
    • CHANGE: (#8418) just ignore a shifter if we cannot get a matching proxy
  • Resources
    • CHANGE: (#8404) improve getReplicas performances by sorting LFNs
    • CHANGE: (#8332) set default time for gfal operations
  • Core
    • CHANGE: (#8419) the weekly index for OpenSearch uses also the year as a prefix

• v9.0.16

  • WMS
    • NEW: (#8422) Support overriding configuration servers in pilot JSON

• v9.0.17

  • WMS
    • NEW: (#8425) Experimental job wrapper for cwl workflows
  • Core
    • CHANGE: (#8435) Support Python 3.14

• v9.0.18

  • Core
    • CHANGE: (#8433) Follow refactoring in DIRACGrid/diracx#741

DiracX

• v0.0.8
  • dense but “technical” release

DiracOS

• 2.58
  • FEAT: (#154) Use pixi for dependency management and build tasks
  • NEW: (#152) fabric and related dependencies for the SSHCE

Dirac-CWL

•  Loris prepared a presentation, attached to the agenda.
  • CWL is there to ultimately replace XML and JDL formats
  • exprimental prototype. CTAO and LHCb on it
  • jobs execution (not submission) via CWL

---

Feature requests, and developers’ issues: inputs and prioritizations from communities

Jobs rescheduling

Do we need to reschedule jobs? Original issue: https://github.com/DIRACGrid/diracx/issues/760. Can we instead resubmit jobs?

•  Federico I asked admins to give their opinion on this. Summaries of the answers I got:
  •  Daniela+Simon
    • if re-submiting the failed job and if then this new job has a new job id, the end user has now lost track of their jobs (replfail & resubmit e.g. once is OK)
    • It would be good to have the error of any failures available, even if it succeeds at the next attempt.
  •  Xiaomei
    • don’t oppose the eventual removal of rescheduling, we need a clear timeline for its complete decommissioning. Its removal must be coordinated with sufficient time for us to properly tune and test new functionalities. It would also be extremely helpful to have concrete examples or guidelines for implementing changes within the transformation system.
  •  Luisa
    • the large majority of rescheduling is due to the JobWrapper not being able to resolve InputData. From what I’ve observed, most of the jobs that can’t access InputData at the first trial, they don’t succeed even after
    • Could:
      • replace automatic rescheduling by automatic resubmission
      • do not support automatic rescheduling but enable the possibility to manually resubmit failed (or selected) jobs, i.e. submit new jobs identical to the selected ones
  •  André
    • https://github.com/DIRACGrid/diracx/issues/760#issuecomment-3876428202
  •  Ueda
    • Attached the following Slides
      • rescheduling is important in several aspects
    • there’s also “resetJob” action. Should that be removed too?
  •  Andrei
    • proposal:
      • Keep rescheduling as it is now but consider it an internal WMS fallback mechanism that can not be accessed by users. “Reset” only for power users.
      • Think of a tool that would “reschedule” input files to generate new jobs.

The discussion will be summarized in the GH issue. Adjurned

RSS in DIRAC, for DiracX

•  Federico I asked admins to answer to a questionnaire about RSS

Possible breaking changes in diracx

• https://github.com/DIRACGrid/diracx/issues/781 would introduce breaking changes. It would only affect already created diracx extensions. @all, would you be affected?
  • –> basically, greeen light as no-one but LHCb runs atm diracx in production

Prioritized backlog: communities input

https://github.com/orgs/DIRACGrid/projects/30/views/3 contains the prioritized backlog.

• objections?
• something from https://github.com/orgs/DIRACGrid/projects/30/views/7 ?

---

AOB

• Certification machines
  • Documentation for how to use them for developments/testing added to …
  •  Federico added the diracx_admin VO.
•  Federico I have been asked to check if we could get https://github.com/solutions/industry/nonprofits. But it’s not possible - we are not a charity or nonprofit in the legal sense.
• CHEP abstract
  • Submitted 2 (?) abstracts:
    • DiracX in action
    • Aligning DIRAC Workflows with CWL: A Unified and Reproducible Workflow Model for Grid-Scale Computing
• DIRAC as an “HSF affiliated project” : https://hepsoftwarefoundation.org/projects/affiliated.html
  • 29th Jan 2026 https://docs.google.com/document/d/1vsdQwMaU36hMb44lrq0og9LvtnWb4fL8x8ZTTYqTUKc/edit?tab=t.0#heading=h.nj23sjpj5u97

---

Next appointments

• Next Dops: 12th March. Topics (draft):

  • CTAO DIRAC+DiracX+Rucio chart
  • Jobs’ match-making (matching) mechanism for DiracX: issues and plans

• WS/hackathons/conferences:

  • DiracX hackathon 24-25 March

10:00 → 11:00 Dirac(X) operations (Dops)

Convener: Federico Stagni (CERN)

note

11:00 → 12:00 Dirac(X) developers (Ddev)

Convener: Alexandre Franck Boyer (CERN)

note

# DIRAC Development Meeting (Ddev)

- **At CERN:** Federico,  Alexandre, Christophe
- **On Zoom:** Alexandre, Ryan, Luisa, Stella, Heloise, Loris, Andrei, Janusz

## Product Goals & Roadmaps

- Transition to DiracX:

```mermaid
flowchart LR
    subgraph CWL["CWL"]
        CWL1("CWL submission endpoint")
        CWL2("CWL production system")
        CWL3("Transformation system machinery"):::blocked
        CWL4("Use CWL natively in new matcher"):::blocked
    end

    subgraph Core["Core"]
        CoreTasks("Tasks")
        Core2("RSS")
        Core3("DMS")
    end

    subgraph WMS["WMS"]
        WMS1("Matcher"):::blocked
        WMS2("Pilot authentication")
        WMS3("Pilot submission"):::blocked
    end

    CWL3 --> CWL4
    CoreTasks --> Core2 --> Core3
    CoreTasks --> WMS1
    CoreTasks --> CWL3
    WMS1 --> CWL4
    CoreTasks --> WMS3

    click CoreTasks "https://www.github.com" "This is a tooltip for a link"

    classDef done fill:#B2DFDB,stroke:#00897B,color:black,stroke-width:2px;
    classDef blocked fill:#BBBBBB,stroke:#222222,color:black,stroke-width:2px;

    subgraph Legend
        L2("Completed"):::done
        L1("Ready for work")
        L3("Blocked"):::blocked
    end
```

- CWL integration:

## Refinements 

### Needs triage
https://github.com/orgs/DIRACGrid/projects/30/views/7

**Goal: build a shared understanding of the project.**

> DIRAC
- [CPUTimeLeft issue](https://github.com/DIRACGrid/DIRAC/issues/8416)
- [Deterministic proxy files for CEs](https://github.com/DIRACGrid/DIRAC/issues/8453)
- [Sub not cleaned when user is removed](https://github.com/DIRACGrid/DIRAC/issues/8191)

> WebAppDIRAC

> diracx
- [Recover lost doc](https://github.com/DIRACGrid/diracx/issues/751)
- [Changelog PR](https://github.com/DIRACGrid/diracx/issues/685)
    - see my latest comment
- [Autorest & mypy](https://github.com/DIRACGrid/diracx/issues/302)
- [Multicore accounting](https://github.com/DIRACGrid/diracx/issues/294)
- [Remove DB types](https://github.com/DIRACGrid/diracx/issues/781)
    - can be safely removed
- [Test that fails, fix it???](https://github.com/DIRACGrid/diracx/pull/583)
    - what's this? just a reproducer: need to become a task
- [renovate update python version :warning:](https://github.com/DIRACGrid/diracx/pull/784)
- [Add tests for job hearbeat and commands](https://github.com/DIRACGrid/diracx/issues/445)
- [Add flexibility on the duration of the refresh token](https://github.com/DIRACGrid/diracx/issues/530)
    - not planned
- [Auth endpoint do not catch errors](https://github.com/DIRACGrid/diracx/issues/559)

> Pilot
- [FillingMode can't be disabled](https://github.com/DIRACGrid/Pilot/issues/275)
- [Python2 drop issue](https://github.com/DIRACGrid/Pilot/issues/276)
    - what do we need exactly here to have a working `dirac.cern.ch` CVMFS repo?

> diracx-charts

> dirac-cwl
- [assigning an output sandbox to a job from the api](https://github.com/DIRACGrid/dirac-cwl/issues/92)

**Should we close these ones (as not planned)?**
> DIRAC
- [Add doc on how to write scripts that interact with DIRAC](https://github.com/DIRACGrid/DIRAC/issues/7115)
    - Does anyone want to write such a documentation? Would it be useful? (I don't think anyone has complained so far)

### [Temporary Section] In progress, predating the new organization

https://github.com/orgs/DIRACGrid/projects/30/views/8

Various people still need to deal with old and staled PRs. We will take them into account in the next sprints. 

> diracx
- [feat (JobDB): pydantic datetime validation](https://github.com/DIRACGrid/diracx/pull/477)
    - Plan to be processed after LHCb Week (Sprint8: Feb 19th - March 5th)
- 

> DIRAC
- [Summary tables](https://github.com/DIRACGrid/DIRAC/pull/8199)

### External dependencies

https://github.com/orgs/DIRACGrid/projects/30/views/9

---

[Planning Poker](https://en.wikipedia.org/wiki/Planning_poker)
Story points values (based on Fibo)
- `1pt`: Trivial, very clear (small bug fix, config change)
- `2pts`: Small, well understood (small feature, clear requirements)
- `3pts`: Medium, some unknowns (moderate feature)
- `5pts`: Large, significant complexity (major feature, integration)
- `8pts`: Very large, many unknowns (should probably be split)
- `13+pts`: TOO BIG - must split!
- `?`: not enough knowledge to answer (remember it's ok to ask any questions)

## Sprints

### Planning (Velocity and Planning Poker)

- Backlog: https://github.com/orgs/DIRACGrid/projects/30/views/3
- Current Sprint: https://github.com/orgs/DIRACGrid/projects/30/views/1


**Average Velocity: 3.07 x FTEs** *Last update: Jan 21st*

#### :warning: Velocity is a planning tool, not a performance target

- Velocity going down is NOT bad
- Velocity going up is NOT always good (might mean over-estimation)
- Velocity varies sprint-to-sprint
- We track it to improve estimation, not to judge people

**What affects velocity:**
- Estimation accuracy (we're still learning)
- Complexity of work

**Our focus:** Delivering value and hitting commitments, not maximizing velocity numbers.

### February 19th (IN PROGRESS):

#### Target and Context
- Chris & Christophe working on the foundations (`diracx-tasks`)
- Clean up existing issues/PRs
- CTAO can submit CWL workflows with input/output data using the RUCIO FC as jobs

#### Availability

- [name=alexandre] 50%
- [name=natthan] 
- [name=luisa] 10%
- [name=loris] 80%
- [name=stella] 80%
- [name=jorge] 80%
- [name=ryan] 20%
- [name=federico] 10% 
- [name=heloise] 50%
- [name=christophe] 0%
- [name=chris] 10% (mostly review)
- [name=janusz] 10%

4 FTEs * 4 = 16 story points

Expected Story Points: 47
Persons: 4
Expected Velocity: 11.7

#### Sprint Planning: 

- Backlog: https://github.com/orgs/DIRACGrid/projects/30/views/3
- Sprint: https://github.com/orgs/DIRACGrid/projects/30/views/1

### February 5th (DONE):

Expected Story Points: 58 
Persons: 3.1
Expected Velocity: 18.7 :warning: 

*29 Story Points / 3.1 people = 9.4 velocity*

Comments:
- LHCb-CERN had a computing workshop (that was known)
- Various people worked on old PRs I did not take into account

#### Sprint review: https://github.com/orgs/DIRACGrid/projects/30/views/11

#### Sprint retrospective

*The sprint is a boat :boat: ; we are trying to reach an island (target); identify anchors (what slowed you down), wind (what helped), and rocks ahead (risks for next sprint)*

:warning: **Focus on the process, not people. We're here to improve together! 🚀**

**:anchor: Anchors (what slowed you down)**
- *Example: Unclear requirements on X; Waiting for Y delayed Z; ...*

**:cloud: Wind (what helped)**
- *Example: Good communication in weekly meetings; Quick code reviews; Clear acceptance criteria on user stories; ...*
- Hackathon did help to get some work done

**🪨 Rocks (risks for next sprint)**
- *Example: Team member K on vacation; Dependency on external API L; Technical debt in M; ...*
- French Holidays

---

### Previous Sprints
#### Summary

- January 21st (DONE):
  - *6 Story Points / 2.5 people = 2.4 velocity*
  - Comments:
    - LHCb-CERN had a team retreat, LHCb-Spain had a conference.

- January 7th:
  - *15 Story Points / 3.9 people = 3.8 velocity*
  - Comments:
    - No specific comment, the sprint was split by the holidays.

- December 10th (DONE):
  - *6 Story Points / 3 people = 2 velocity*
  - Comments:
    - About the same as the previous sprint: still a gap between expected/actual availability

- November 26th (DONE):
  - *6 Story Points / 3 people = 2 velocity*
  - Comments:
    - Much lower than the previous sprint because it included tasks started before the sprint.
    - Lots of "almost done" PRs: we are improving the description of the tasks and their size but still not enough (each task should bring value though).

- November 10th (DONE):
  - *22 Story Points / 4.3 people = 5.1 velocity*

#### Actionable Results from the Retrospective

- **Action:** Better communicate when a PR is going to be big, as soon as possible. Split the work in this case.
  - Owner: developers
  - When: Sprint6
  - Status: 21/01/26 in progress
- **Action:** Better use of the mattermost channel to get reviews on a given PR
  - Owner: everyone
  - By when: Sprint3
  - Status: 04/02/26 DONE
- **Action:** Define estimates and velocity based on Sprint2's results, taking into account external contributions (bonus Story Points) and availability
  - Owner: alexandre
  - By when: Sprint3
  - Status: DONE
- **Action:** Better define the scrum roles
  - Owner: alexandre
  - By when: Sprint5
  - Status: DONE
- **Action:** Better define `DONE` criteria (what should be included into the PR, and how to make sure we are not introducing too much technical debt)
  - Owner: everyone
  - By when: Sprint2
  - Status: DONE
- **Action:** Avoid planning dependent tasks in a same sprint
  - Owner: everyone
  - By when: Sprint2
  - Status: DONE

## AOB