Choose timezone
Your profile timezone:
Ddev
Description
Live meeting notes: https://codimd.web.cern.ch/QMVdoWxiShS9Iv9Mp9ifKw?view
BiLD
# DIRAC Development Meeting (Ddev)
**At CERN**: Alan, Alexander, Pantellis, Juraj, Benedikt, Alexandre, Chris
**On Zoom**: Bertrand, Dhiraj, Hideki, Heloise, Jorge, Ryun, Francesco, Janusz
**Apologies**: Christophe, Federico, Marco, Natthan
## Product Goals & Roadmaps
## Refinements
### Needs triage
https://github.com/orgs/DIRACGrid/projects/30/views/7
**Goal: build a shared understanding of the project.**
> .github
> DIRAC
- [Add getFileUserMetadataBulk to the DFC](https://github.com/DIRACGrid/DIRAC/issues/8826)
> DiracOS2
> WebAppDIRAC
> diracx-web
> diracx
- [Possible DB connection leaks](https://github.com/DIRACGrid/diracx/issues/1129)
- [Documentation about the DB schemas](https://github.com/DIRACGrid/diracx/issues/1100)
- [DiracX task to make sure AuthDB is healthy](https://github.com/DIRACGrid/diracx/issues/1066)
- [What is a system in DiracX](https://github.com/DIRACGrid/diracx/issues/1169)
- [DB Migration](https://github.com/DIRACGrid/diracx/issues/982) - to elaborate
- [Discussion about PRs](https://github.com/DIRACGrid/diracx/discussions/1155)
- [Discussion about AI Agents](https://github.com/DIRACGrid/diracx/discussions/1156)
> Pilot
> diracx-charts
> dirac-cwl
> signurlarity
**External deps**
### [Temporary Section] In progress, predating the new organization
https://github.com/orgs/DIRACGrid/projects/30/views/8
Various people still need to deal with old and staled PRs. We will take them into account in the next sprints.
### External dependencies
https://github.com/orgs/DIRACGrid/projects/30/views/9
TODO: set up a meeting for diracx-charts discussion 1047
TODO: add more details about the factory settings in #1100
TODO: investigate the "review team" concept in github
---
[Planning Poker](https://en.wikipedia.org/wiki/Planning_poker)
Story points estimate the effort to get an item **merged**: design, code, tests, docs **and review**.
- `1pt`: Trivial, very clear (small bug fix, config change). One small PR, reviewed in minutes.
- `2pts`: Small, well understood (small feature, clear requirements). One PR, easy to review.
- `3pts`: Medium, some unknowns (moderate feature). One or two PRs, each easy to review in one sitting.
- `5pts`: Large, significant complexity (major feature, integration). Most of a person's sprint; several PRs; interfaces agreed on the issue before coding.
- `8pts`: Too large for one sprint: **split before starting**.
- `13+pts`: An epic, not an issue: split into issues.
- `?`: not enough knowledge to answer (remember it's ok to ask any questions)
## Sprints
### Planning (Velocity and Planning Poker)
- Backlog: https://github.com/orgs/DIRACGrid/projects/30/views/3
- Current Sprint: https://github.com/orgs/DIRACGrid/projects/30/views/1
**Average Velocity: ~5.8 x FTEs** *Last update: Jun 25th*
#### :warning: Velocity is a planning tool, not a performance target
- Velocity going down is NOT bad
- Velocity going up is NOT always good (might mean over-estimation)
- Velocity varies sprint-to-sprint
- We track it to improve estimation, not to judge people
**What affects velocity:**
- Estimation accuracy (we're still learning)
- Complexity of work
**Our focus:** Delivering value and hitting commitments, not maximizing velocity numbers.
### November 5th (IN PROGRESS):
#### Target and Context
- **Transformation System**
- Able to run a simple workgraph (only happy path) (2/2)
- **Transition**:
Clean up existing issues/PRs: [Burning Charts](https://github.com/orgs/DIRACGrid/projects/30/insights?period=3M)
- **RSS**: Finish Phase2
- **Compute Backend**:
- `intercede`: first integration test
#### Availability
- [name=Alexandre] 50%
- [name=Natthan] %
- [name=Luisa] %
- [name=Loris] 20%
- [name=Stella] %
- [name=Jorge] 10%
- [name=Ryun] 30%
- [name=Federico] 10%
- [name=Héloïse] 30%
- [name=Christophe] 10%
- [name=Chris] 10%
- [name=Janusz] 10%
- [name=Mazen] 10%
- [name=Andrei] %
- [name=Simon] %
- [name=Daniela] %
- [name=Hideki] %
- [name=Benedikt] 20%
- [name=Juraj] 10%
- [name=Todor] 10%
- [name=Henryk] %
- [name=Dhiraj] 10%
- [name=Vijay] 10%
- [name=Andrea] %
- [name=Alan] 50%
- [name=Camille] %
- [name=Marco] 10%
- [name=Valentin] 10%
- [name=Alexander] 100%
- [name=Francesco] 10%
- [name=Panteli] %
- [name=Quinn] 10%
_ FTEs * _ = _ story points
Expected Story Points:
Persons:
Expected Velocity:
#### Sprint Planning:
- Backlog: https://github.com/orgs/DIRACGrid/projects/30/views/3
- Sprint: https://github.com/orgs/DIRACGrid/projects/30/views/1
### October 29th (DONE):
Expected Story Points: 49
Persons: 3.8
Expected Velocity: 12.9
33 / 3.8 = 8.7
Comments: NTR
#### Sprint review: https://github.com/orgs/DIRACGrid/projects/30/views/11
Related to our goals:
- **Transformation System**
- NTR
- **RSS**
- Simplified RSS reading part in DIRAC (2 status only: Active and Banned)
- **DiracX Compute Backend**
- 1st version of intercede POC: with a Fake Backend
- Pilot migration: paused for now
- **DIRAC maintenance:**
- Mostly bug fixes + improving performances
#### Sprint retrospective
*The sprint is a boat :boat: ; we are trying to reach an island (target); identify anchors (what slowed you down), wind (what helped), and rocks ahead (risks for next sprint)*
:warning: **Focus on the process, not people. We're here to improve together! 🚀**
**:anchor: Anchors (what slowed you down)**
- *Example: Unclear requirements on X; Waiting for Y delayed Z; ...*
- Rebasing PRs: a lot of renovate PRs merged forcing us to rebased our PRs very often, it's painful. The reviewer has to click on "rebase PR" button (unless there is a conflict)
- A lot of Renovate PRs: someone has to do the job:
- either randomly chosen during the beginning of a sprint
- or we investigate this "agent" tab in github, seeing if we can use it here
- a broader discussion would be needed about the use of LLMs/agents within DIRACGrid (Federico will open it)
- having everyone involved in the review process
**:cloud: Wind (what helped)**
- *Example: Good communication in weekly meetings; Quick code reviews; Clear acceptance criteria on user stories; ...*
**🪨 Rocks (risks for next sprint)**
- *Example: Team member K on vacation; Dependency on external API L; Technical debt in M; ...*
- How to solve conflicts among the development we have (e.g. Transformation System PRs)
- 3 points for feeder and packer were too low (a lot of code to review): see if we are using the story points correctly - writing code is simpler, review is more complex, it should be taken into account
---
### Previous Sprints
#### Summary
- October 15th:
- *17 Story Points / 3.8 people = 4.5 velocity*
- Comments: Preparation of the DIRAC User Workshop
- October 1st:
- *13 Story Points / 2.1 people = 6 velocity*
- Comments: Preparation of the DIRAC User Workshop
- September 17th:
- *35 Story Points / 3.3 people = 10 velocity*
- Comments: Preparation of the DIRAC User Workshop + CHEP Papers + CMS Computing Workshop
- September 3rd:
- *18 Story Points / 2.3 people = 7.8 velocity*
- Comments: LHCb Week + many urgent fixes on DIRAC slowed down the main reviewers
- August 20th:
- *7 Story Points / 2.5 people = 2.8 velocity*
- Comments: too many items given the number of people, but still a few items going from 1 sprint to another (mostly due to holidays here)
- August 6th:
- *22 Story Points / 3.1 people = 7.1 velocity*
- Comments: almost managed to reach the expected velocity, but the defined goal are not yet reached, probably too many. Seems like we better plan when everyone is on holidays :slightly_smiling_face:
- July 23rd:
- *19 Story Points / 3.7 people = 5.1 velocity*
- Comments: NTR
- July 23rd:
- *19 Story Points / 3.7 people = 5.1 velocity*
- Comments: NTR
- July 9th:
- *27 Story Points / 4.3 people = 6.3 velocity*
- Comments:
- A lot of people ramping up since the hackathon, a lot of invisible design work under the hood that does not translate into PR merged for now.
- June 25th:
- *18 Story Points / 4.2 people = 4.3 velocity*
- Comments:
- As usual, we are still overestimating our efforts. A lot of unexpected bug fixes preventing us from going forward with current items.
- June 11th:
- *20 Story Points / 3.8 people = 5.2 velocity*
- Comments:
- Big difference between expected velocity and effective one.
- May 28th:
- *16.2 Story Points / 4.2 people = 3.8 velocity*
- Comments:
- Big difference between expected velocity and effective one.
- CHEP planning and long weekends (CH, France) largely affected it.
- May 14th:
- *11 Story Points / 4 people = 2.75 velocity*
- Comments:
- Lowest velocity since we started, but:
- RSS end of phase1 is actually trickier than what we initially thought
- Many people having to start preparing presentations (CHEP, LPC retreat)
- Many people are working on tasks that are not in the scope of the sprint (to prepare future sprints): LHCb commands to replace the workflow modules, integration of CWL job submission endpoint within diracx
- A CI failure in DIRAC preventing from merging
- April 30th:
- *42 Story Points / 4.1 people = 10.2 velocity*
- Comments:
- ~1/4 of the counted SP come from the integration of the tasks
- `diracx-tasks` are here :tada:
- All the essential components are here to transition now.
- RSS Phase1:
- Should have been completed but it's still under development. [name=Loris] any blocking point?
- New Matcher:
- working on a v0.2 schema design expliciting more details about what we want
- April 16th:
- *20 Story Points / 2.6 people = 7.7 velocity*
- Comments:
- Less people available during the sprint (holidays, CTAO had deadlines). Also some people seemed to spend more time than originally described, some of them less time.
- A lot of bug fixes that were not planned
- April 2nd:
- *37 Story Points / 3.1 people = 11.9 velocity*
- Comments: NTR
- March 19th:
- *23 Story Points / 3.5 people = 7.2 velocity*
- Comments:
- Need to adapt the velocity computation because we are processing a lot of tasks not planned originally in the sprint (which is expected since we still have a lot of PRs without any attached issue to process, ...)
- March 5th:
- *19 Story Points / 2.8 people = 6.8 velocity*
- Comments:
- Less people available during this sprint, but more realistic expectation, we almost reached the expected velocity!!
- LHCb AI hackathon: [name=Alexandre] was much less available than expected.
- Took into account items that were in progress before scrum process (added some SP): resurrecting diracx-web, RSS simplified...
- February 19th:
- *38 Story Points / 4.4 people = 8.6 velocity*
- Comments:
- French holidays
- [name=Alexandre] was more available than expected, but did not manage to quickly follow all the PRs.
- A few tasks have been delayed (10 SP): waiting for further discussion on scheduling and diagrams for new LHCb workflows
- Lot of "unplanned" items: expected as long as we have to deal with the large backlog of old items.
- February 5th:
- *29 Story Points / 3.1 people = 9.4 velocity*
- Comments:
- LHCb-CERN had a computing workshop
- Various people worked on old PRs I did not take into account :warning:
- January 21st:
- *6 Story Points / 2.5 people = 2.4 velocity*
- Comments:
- LHCb-CERN had a team retreat, LHCb-Spain had a conference.
- January 7th:
- *15 Story Points / 3.9 people = 3.8 velocity*
- Comments:
- No specific comment, the sprint was split by the holidays.
- December 10th:
- *6 Story Points / 3 people = 2 velocity*
- Comments:
- About the same as the previous sprint: still a gap between expected/actual availability
- November 26th:
- *6 Story Points / 3 people = 2 velocity*
- Comments:
- Much lower than the previous sprint because it included tasks started before the sprint.
- Lots of "almost done" PRs: we are improving the description of the tasks and their size but still not enough (each task should bring value though).
- November 10th:
- *22 Story Points / 4.3 people = 5.1 velocity*
#### Actionable Results from the Retrospective
> may be we should track the technical debt (have a metric about that)
- **Action:** Developers should start reviewing PRs if they have some time available and they finished their tasks (at least the first passes)
- Owner: developers
- When: Sprint17
- Status: 11/06/26 In Progress
- **Action:** Share a more detailed roadmap that we should all understand.
- Owner: architects
- When: Sprint17
- Status: 11/06/26 In Progress
- **Action:** Reviewer has to double check titles of the PRs before merging.
- Owner: reviewers
- When: Sprint16
- Status: 11/06/26 In Progress
- **Action:** Feature PRs should be thoroughly tested in certification.
- Owner: developers
- When: Sprint12
- Status: 29/04/26 In Progress
- **Action:** Avoid verbose (AI-generated) issues with many implementation details that can deprecate over time.
- Owner: developers and product owners
- When: Sprint11
- Status: 15/04/26 In Progress
- **Action:** Better view of the PRs ready to be reviewed vs needing changes.
- Owner: developers
- When: Sprint8
- Status: 15/04/26 In Progress
- **Action:** Better communicate when a PR is going to be big, as soon as possible. Split the work in this case.
- Owner: developers
- When: Sprint6
- Status: 21/01/26 DONE
- **Action:** Better use of the mattermost channel to get reviews on a given PR
- Owner: everyone
- By when: Sprint3
- Status: 04/02/26 DONE
- **Action:** Define estimates and velocity based on Sprint2's results, taking into account external contributions (bonus Story Points) and availability
- Owner: alexandre
- By when: Sprint3
- Status: DONE
- **Action:** Better define the scrum roles
- Owner: alexandre
- By when: Sprint5
- Status: DONE
- **Action:** Better define `DONE` criteria (what should be included into the PR, and how to make sure we are not introducing too much technical debt)
- Owner: everyone
- By when: Sprint2
- Status: DONE
- **Action:** Avoid planning dependent tasks in a same sprint
- Owner: everyone
- By when: Sprint2
- Status: DONE
## AOB
Slurm: new version
- changed behaviour
```
Hi folks, just a heads up about something we found out recently at CMS that might also affect DIRAC pilots. A recent Slurm version had a change in behavior that caused issues for CMS pilots in at least one site (see the [original discussion](https://helpdesk.ggus.eu/#ticket/zoom/1004044)).
Slurm 26.05.0, released in May 2026 introduces the following [change](https://github.com/SchedMD/slurm/blob/490ce34caf2ea122ab85b926b4b36f0f56302b8e/CHANGELOG/slurm-26.05.md?plain=1#L72):
> Set max_nodes from num_tasks when not explicitly set
CMS pilots do not explicitly set max_nodes or num_tasks on their requests to the CE. This caused the max_nodes parameter to change after the Slurm update.
After 26.05.0, the pilots reach Slurm with the following parameters:
NumNodes=1-8 NumCPUs=8 NumTasks=8 CPUs/Task=1
While I expected to see something like this:
NumNodes=1 NumCPUs=8 NumTasks=1 CPUs/Task=8
The result of the change is that this particular site stopped running CMS pilots because the Slurm queue only accepted jobs with NumNodes=1. The site admin did a workaround to allow jobs with NumNodes=1-8 in the queue, but IMO this is not a definitive solution. The semantics of a pilot (at least for CMS) is a single task running in a single node, it does not make sense to divide and distribute a pilot across several nodes. For the moment the workaround works, but our goal is to have the num_tasks parameter set to 1 explicitly on the client (GlideinWMS).
So far I believe the grid middlewares CMS uses (ARC and HTCondor-CE) do not expose this parameter to the client. So there is work to be done on the middlewares to allow that.
Check if you are setting the num_tasks parameter explicitly when interacting with Slurm sites. If not, you might bump into similar issues as sites start upgrading to Slurm 26.05.x.
```
There are minutes attached to this event.
Show them.