DiracX WP: Transformation System

Name: DiracX WP: Transformation System
Start: 2026-10-30T10:00:00+01:00
End: 2026-10-30T11:00:00+01:00
Location: CERN

Friday 30 Oct 2026, 10:00 → 11:00 Europe/Zurich

2/R-014 (CERN)

2/R-014

CERN

Show room on map

Description

Discussions around the Transformation System: https://codimd.web.cern.ch/SVAX3vHySq6mhbufbp87iQ

62504856418

Federico Stagni

Join via phone

Hide

# Transformation System: planning meeting, 30 Oct 2026

[TOC]

**Links**
- Epic: [#1092 Transformation System](https://github.com/DIRACGrid/diracx/issues/1092)
- Deliverable 1: [#1081](https://github.com/DIRACGrid/diracx/issues/1081)
- Project board: [DIRAC development planning](https://github.com/orgs/DIRACGrid/projects/30)

## Goals & Rules (5 min)

### Goals

1. Understand what slowed us in sprint 1 and agree how we slice work from now on. **(10 min)**
2. Deliverable 1: owners, review order, design decisions, and what "done" looks like. **(30 min)**
3. Deliverable 2: agree the frame and the homework; we define it together at the next meeting. **(10 min)**

Issues are written **after** the meeting, from this document.

### Rules of the meeting

- **Timeboxes are strict.** The facilitator may stop a discussion. That does not mean it is unimportant: it goes to the [Parking lot](#Parking-lot) with an owner.
- **Decisions here, details in issues.** We decide *what* and *who*; *how* goes into the issue or the PR.
- **Write in this document freely** in the sections marked ✍️.
- **A decision is taken when nobody has an objection they cannot live with.** If we cannot decide in the timebox, the question becomes a short experiment (spike), with an owner and a date.
- **No blame.** We look at how we work, not at who did what.

## Agenda (60 min)

| Time | Topic |
|---|---|
| 0–5 | Goals and rules |
| 5–15 | What slowed us in sprint 1, and how we slice work |
| 15–45 | Deliverable 1: owners, review order, design decisions, "done" |
| 45–55 | Deliverable 2: the frame and the homework |
| 55–60 | Read back: decisions, owners, reviewers, who writes which issue |

## 1. What slowed us in sprint 1 (10 min)

Facts (30 Oct, morning):
- **15 Transformation System PRs are open, ~19,800 added lines in total.** Outside the feeder stack, most are ~1,000 to 4,100 lines; the largest is
[#1119](https://github.com/DIRACGrid/diracx/pull/1119) (4,145).
- **12 of them have no review yet**; the packer [#1102](https://github.com/DIRACGrid/diracx/pull/1102) has 32 reviews.
- **Several design questions are being settled inside PR reviews (packer drain semantics, JSON hashing, plugin naming, task locks).**
- Some already became issues ([#1139](https://github.com/DIRACGrid/diracx/issues/1139), [#1140](https://github.com/DIRACGrid/diracx/issues/1140),
[#1142](https://github.com/DIRACGrid/diracx/issues/1142)).
- Most PRs were opened in the last week of sprint 25 (21–26 Oct).
- This morning the feeder was re-split into 5 stacked PRs ([#1144](https://github.com/DIRACGrid/diracx/pull/1144) → [#1145](https://github.com/DIRACGrid/diracx/pull/1145) →
[#1146](https://github.com/DIRACGrid/diracx/pull/1146) → [#1147](https://github.com/DIRACGrid/diracx/pull/1147) → [#1148](https://github.com/DIRACGrid/diracx/pull/1148)), 100 to 700 new
lines each: the direction we want. They all target `main`, so each one also shows the PRs below it (the docs PR shows +1,753).

How we slice work from now on (to confirm):
- An issue is a piece of value with one owner; it can have several PRs.
- A PR is small enough to review in one sitting (~300–500 lines). First PR: the interface or a thin working version.
- A design question found during review that affects other components becomes its own issue (see [Parking lot](#Parking-lot) and [Design decisions](#Design-decisions-Deliverable-1)). It needs discussion (Mattermost channel?) to see where it fits in the tree of issues.

**Any comment?**

- Marco: hard to know in advance, become messy over time: that's why I created a claude artifact
- Chris: fine to split once the PR is becoming too big
- Alex: as a reviewer, we should may be not start reviewing the PR if it's too big (to avoid getting comments, then we don't want to lose track of them)
- Alan: squashing and rewrite the commit make sense
- Chris: I never remember my previous comments, so it's probably fine if we keep track of some of them

Marco will close his PR, and will open multiple smaller PRs

- Alan: current issues are ok (if we heavily rely on AI), otherwise I would have been in favor of smaller tasks. If we can break down into smaller tasks, it can help us understand the system (e.g. REST API). Might be easier.
- Federico: sometimes, you just need to have a big PR
- Chris: it should be considered case by case (just guidelines)

## 2. Deliverable 1 (30 min)

**Goal ([#1081](https://github.com/DIRACGrid/diracx/issues/1081)):** process a basic workgraph made of a single transformation, in the diracx-charts dev environment.

**"Done" (proposal):** in the diracx-charts dev environment, submit a workgraph with one transformation → the feeder creates N inputs → the packer makes parcels → the dummy backend runs them → the counters show N done.

**Board**: https://github.com/orgs/DIRACGrid/projects/30/views/13

**Architecture: who reads and writes what**

```mermaid
flowchart LR
subgraph SUB["Submission (diracx-routers + diracx-logic)"]
submit["Submit a workgraph #1086"]
ingest["CWL ingestion #1080"]
end

subgraph TASKS["diracx-tasks: periodic tasks, each *_ENABLED, off by default"]
feeder["Feeder task #1082 lock: feeder mutex + shared status lock"]
packer["Packer task #1083 lock: packer mutex + shared status lock"]
monitor["Dummy backend monitor #1084"]
complete["Dummy completion task one per parcel"]
end

subgraph PLUGINS["Plugins propose, the core writes"]
fplug["Feeder plugin yields Input, SetFeederState, NoMoreInputs"]
pplug["Packer plugin: Trivial / FixedSize yields ComputeParcel, BadInput, DelayInput"]
end

subgraph DB["TransformationDB"]
tr[("Transformations Status, Feeder, FeederArgs, FeederState, FeederEnabled, Packer, PackerArgs")]
ct[("ComputeTransformations ProcessID, RequirementsTemplateID")]
proc[("Processes")]
req[("RequirementsTemplates")]
inp[("Inputs Unassigned → Assigned → Processed")]
par[("Parcels + ComputeParcels + ParcelInputs Unassigned → Reserved → Assigned → Completing → Done")]
cnt[("Counters journal #1085")]
end

submit -- "store process" --> ingest --> proc
submit -- "workgraph and transformation rows" --> tr
submit --> ct
ct -.-> proc
ct -.-> req

feeder -- "1. read config and state" --> tr
feeder --- fplug
feeder -- "2. insert Unassigned, dedup by hash" --> inp
feeder -- "save state, disable when no more inputs" --> tr

packer -- "3. claim Unassigned, SKIP LOCKED" --> inp
packer --- pplug
packer -- "4. create parcels, inputs → Assigned" --> par
packer -- "merged requirements" --> req

monitor -- "5. claim Unassigned parcels, Backend = dummy" --> par
monitor -- "schedule" --> complete
complete -- "6. → Done, inputs → Processed" --> par

feeder -. "journal" .-> cnt
packer -. "journal" .-> cnt
complete -. "journal" .-> cnt
```

**Shortcut to discuss:** for Deliverable 1, [#1086](https://github.com/DIRACGrid/diracx/issues/1086) stores the process without deduplication or policy checks; [#1110](https://github.com/DIRACGrid/diracx/issues/1110) plugs in when it merges.

## Design decisions (Deliverable 1)

Collected from the issues and PR reviews (29 Oct). For each: decide now, or name **who decides by when**.

- **D1 · How do we store CWL processes?** ([#1080](https://github.com/DIRACGrid/diracx/issues/1080), [#1110](https://github.com/DIRACGrid/diracx/issues/1110))
- Context: each transformation runs a CWL process: a CommandLineTool (e.g. Gauss) or a sub-workflow made of several CommandLineTools (e.g. Boole then Moore). Processes are stored in `Processes`.
- Options: **one row per transformation**, everything it runs inlined ([#1110](https://github.com/DIRACGrid/diracx/issues/1110): one read per parcel, but a shared CommandLineTool is stored several times) / **one row per CommandLineTool and per workflow**, workflows pointing to their children by content ID ([#1080](https://github.com/DIRACGrid/diracx/issues/1080)'s DoD: each one stored once, but reading what a parcel runs follows the references).
- Decision:
- Benedikt: but we will have some duplication
- Chris: it does not make too much sense to split further than the job level
- Chris: a parcel can potentially run multiple CLT (and steps).
- Federico: reproducibility?
- **Final decision**: one row per transformation

- **D2 · Does submission wait for full CWL ingestion?** ([#1086](https://github.com/DIRACGrid/diracx/issues/1086))
- Context: submission must point each transformation to a stored process. Full ingestion (dedup, validation, policy) is 4 PRs in a row ([#1106](https://github.com/DIRACGrid/diracx/issues/1106) → [#1107](https://github.com/DIRACGrid/diracx/issues/1107) + [#1109](https://github.com/DIRACGrid/diracx/issues/1109) → [#1110](https://github.com/DIRACGrid/diracx/issues/1110)), ~7k lines to review.
- Options: **wait** for the whole chain / **shortcut**: for D1, store the process as submitted, without validation (in terms of DiracX validation); plug [#1110](https://github.com/DIRACGrid/diracx/issues/1110) in when it merges.
- Decision:
- Chris: deduplication will happen naturally. You just assume that the document is already valid (this should be 2 separate issues)
- **Final decision**: shortcut (already defined during workshop)

- **D4 · DB unit tests on MySQL/MariaDB, not only SQLite** ([#1108](https://github.com/DIRACGrid/diracx/issues/1108))
- Context: unit tests run on SQLite, production on MySQL/MariaDB, which behave differently (JSON columns, locks). [#1115](https://github.com/DIRACGrid/diracx/pull/1115) adds real servers in CI for the transformation DB only; the review asks "why not every DB?".
- Options: **transformation DB only** (as in [#1115](https://github.com/DIRACGrid/diracx/pull/1115)) / **every DB**, one shared fixture, servers from conda-forge.
- Decision:
- Chris: we already have this with OpenSearch.
- Federico: every DB is necessary anyway
- Chris: spawning a server in the ga (difficult to run by hand). Should we do it within run_demo?
- Christophe: already happening in diracx-charts (run unit-tests) -> a lot of plumbing already there. That shouldn't that complex to do
- Chris: either start unit tests using run_demo, or using conda-forge: the run_demo approach likely better
- Alan: if it's too costly, can only run with tagged version
- Federico: we had several issues in the past
- Chris: we should try to run it as quickly as we can
- **Final decision**: try the run_demo approach. Ryun will close his PR, and make his issue more generic

- **D5 · One way to hash JSON** ([#1139](https://github.com/DIRACGrid/diracx/issues/1139), [#1106](https://github.com/DIRACGrid/diracx/issues/1106))
- Context: some tables store each document once, keyed by its hash. The packer and CWL ingestion hash JSON slightly differently, so the same document could get two IDs and be stored twice, silently.
- Options: **one shared helper** in diracx-core for every writer (which one? owner?) / **keep one function per table**. The feeder's input hash (xxh3) has another purpose and can stay.
- Decision:
- Ryun: but this is different from CWL
- Chris: but it's all JSON blob to hash
- **Final decision**: PR needs to be created and merged today, then other PRs should be rebased upon it

- **D6 · Which packers ship first?** ([#1140](https://github.com/DIRACGrid/diracx/issues/1140), [#1102](https://github.com/DIRACGrid/diracx/pull/1102))
- Context: [#1102](https://github.com/DIRACGrid/diracx/pull/1102) ships `FixedSize` (groups of N inputs; holds back a smaller last group while the feeder still runs). `Trivial` (one parcel per input, nothing held back) is enough for the D1 demo.
- Options: **Trivial first**, FixedSize later / **both** now. Open point: a transformation without a feeder counts as "feeding done" (DX-ADR-005), so FixedSize packs the remainder at once. Do inputs ever arrive without a feeder?
- Decision:
- Federico: what if we put the trivial one wihtin gubbins to test the extensibility
- **Final decision**: both, but `Trivial` one goes into gubbins
- Alan: in the same mindset, another ticket to have an extension feeder in gubbins. Alan will create the ticket and bind it to the `Feeder` ticket

- **D7 · Where the code goes and how it is named** ([#1102](https://github.com/DIRACGrid/diracx/pull/1102), [#1130](https://github.com/DIRACGrid/diracx/pull/1130), [#1144](https://github.com/DIRACGrid/diracx/pull/1144)–[#1147](https://github.com/DIRACGrid/diracx/pull/1147))
- Context: the packer, feeder and dummy backend PRs each picked their own layout. Agreeing once avoids renames during review.
- Proposal: everything under `transformation[s]/`, one module per plugin type (packers, feeders, hooks, actions); `logic/transformation/packer.py` next to `logic/transformation/feeder.py` (or in `core/models`?); dummy backend in `tasks/compute_backends/dummy/`;
- Decision:
- Chris: we don't necessarily have a clear vision how that should be, but this will come once we have the first implementation
- Francesco: we can move components if needed
- **Final decision**: it goes case by case and we structure along the way

- **D8 · Locks between feeder, packer and status changes** ([#1082](https://github.com/DIRACGrid/diracx/issues/1082), [#1142](https://github.com/DIRACGrid/diracx/issues/1142))
- Context: feeder and packer may run at the same time, but not while a transformation is being paused or finalised. Agreed on [#1102](https://github.com/DIRACGrid/diracx/pull/1102): each task holds its own lock plus a shared "status" lock; pausing or finalising takes the status lock exclusively.
- Open: does the new feeder ([#1147](https://github.com/DIRACGrid/diracx/pull/1147)) do the same? [#1142](https://github.com/DIRACGrid/diracx/issues/1142): if a worker crashes, the shared lock is never released and the transformation can never be paused again. Fix before pause/finalise exist?
- Decision:

## 3. Deliverable 2: frame and homework (10 min)

**Frame:** a workgraph with at least two transformations, where the outputs of the first feed the second (e.g. MCSimulation → MCReconstruction from [#1080](https://github.com/DIRACGrid/diracx/issues/1080)).

**Homework** (≈ 30 min, before next meeting): write your answers under your name below. Short bullets are fine; reading the others' answers first is fine too.

- **Q1** What is the smallest demo that proves T1's outputs feed T2? (one sentence)
- **Q2** Who records a parcel's outputs, and when? (job wrapper, backend, separate task; on success only?)
- **Q3** Where do the outputs live? (file catalogue, Rucio, TransformationDB)
- **Q4** What needs an ADR (a job wrapper equivalent?), and what can be a one-sprint experiment?
- **Q5** What is out of scope for Deliverable 2?
- **Q6** What worries you most? (risk, unknown, missing piece)

Before the next meeting, the facilitator groups the answers and posts a summary here. The meeting then discusses the disagreements only.

✍️ **Answers** (copy a block for yourself)

- **Name:**
- Q1:
- Q2:
- Q3:
- Q4:
- Q5:
- Q6:
- **Name:**
- Q1:
- Q2:
- Q3:
- Q4:
- Q5:
- Q6:

**Summary**
- Agreed:
- Disagreements to discuss:
- Open questions → spike:

## Parking lot

✍️ Anything important that is not for today. Add an owner and a next step (issue, ddev meeting, next planning).

| Topic | Raised by | Owner | Next step |
|---|---|---|---|
| [#1129](https://github.com/DIRACGrid/diracx/issues/1129) DB connection leak (core diracx) | Francesco |Alexandre | Discussed and refined in next Ddev |
| Shared interface to load the packer/feeder plugins | Alan | Alexandre | Discussed once Packer and Feeder land in `main` |
| | | | |

## Actions

| What | Who | When |
| ------------------------------------- | --------- | ---------------------------------- |
| D1: adjust the issue | Alexandre | After the meeting ==Done== |
| D4: adjust the issue and close the PR | Ryun | After the meeting ==Done== |
| D5: open PR to provide shared hashing functions | Alan | After the meeting ==Done== |
| D6: add the `TrivialPacker` within gubbins | Marco | After the meeting |
| Split the packer PR into several smaller PRs | Marco | After the meeting ==Done== |
| Deliverable 2 homework | everyone | next meeting: same time next week |
| Summarise the Deliverable 2 answers | | after next meeting |
| | | |

There are minutes attached to this event. Show them.

The agenda of this meeting is empty