BiLD – 13/11/2025

At CERN: Federico, Christophe, Christopher, Ryun, Alexandre
On Zoom: Ueda, Daniela, Hideki, Alexei, Natthan, Andrei, Mazen, Jorge, Xiaomei, Dhiraj
Apologies:

Previous meetings

Last BiLD was 3 weeks ago

DIRAC communities roundtable

LHCb:

Federico+Christopher+Christophe+Alexandre+Ryun

Updated to proper (LHCb)DIRAC and (lhcb)DiracX releases
We finally solved issues with CS becoming unavailable
- “solved” by having a load balancer (external to DIRAC) in front of the CS slaves. Just a machine installing HAProxy. DIRAC’s balancer exposes issues when there’s a high load
  - Andrei+Xiaomei saw in Juno similar issue, maybe this could be applied there too

Belle2

Hideki, Ueda, Dhiraj

Scalability issues: MySQL. Probably caused by many short jobs
- Andrei there are plots for jobs/second executed that could be useful for you
- there’s an overhead for starting jobs so should avoid running 1-minute jobs
Switching VOMS, also KEK CAs.
from previous meeting Found some stomp related issues, will check if already signaled, if not will create a new one

Juno+BES3:

Xiaomei

Some bottleneck issues lately, many small jobs hammering the CS and MySQL, and the disk where Sandboxes are located
- Andrei added few CS slaves
- For Sandboxes --> the main way forward is use DiracX and S3 for sandbox store
Forced to use Multi-hop. Mostly successfull, but issues reported
- Christophe saw the issues, will look at it soon

EGI

Andrei

Smooth running. Chasing sites to update the configuration (VOMS, tokens, etc)
Created GreenDIRAC extension for keeping related developments: GreenReportingAgent, energy consumption by jobs tool (Mazen)
- Submitting energy metrics for each job into the DB
Started a GreenSiteDirector
- work in progress, for submitting to sites that have a lower energy footprint
Got functional D9/Dx0.0.1 certification installation (Bertrand):
- DIRAC9 host + 2 hosts k3s cluster for DiracX + S3/minIO
  - Chris+Federico should not run on 2. Better 1 or 3
- Opensearch with certificate authentication
  - will contribute to DiracX for this
- IAM dev IdP, failed with dteam IAM, Check-In in progress
- wants to add also a MQ

GridPP:

Daniela, Simon

No change in production.
DIRACx investigation slow, mainly due to Daniela being pre-occupied with other things.
Could we at some point in the meeting have a quick look at: https://github.com/DIRACGrid/diracx/discussions/700 (“CS repo directory problems in container deployment”)

Topics from GitHub discussions and bots

only un-answered DIRAC and DiracX topics with discussion updates:
- NTR

Releases

DIRAC

v8.0.76
- no new releases since July
v9.0.6
- Configuration
  - CHANGE: (#8353) add PreferredURLPatterns to pilot.json
- FrameworkSystem
  - NEW: (#8328) allow for multiple client per provider
- DataManagementSystem
  - NEW: (#8328) FTS3 gets tokens directly

Issues (still there?):

from previous meeting https://github.com/conda-forge/dirac-grid-feedstock/issues/170 https://github.com/conda-forge/dirac-grid-feedstock/issues/169
- cburr will take care of these ASAP

DiracX

v0.0.2
- created 3 weeks ago

Release planning, tests and certification

Shall we use immutable releases?
- done for DIRAC and DiracX
Upgrade to v9+0.0.1:
- Federico I have not touched the guide in the last 2 or 3 weeks, any more advise? Anyone else tried it out?
Certification machines
- Federico I have been asked to add CMS VO there, with just 1 user.
  * dteam will be enough
  - Andrei CMS could go in EGI, even
Next hackathon(s)
- Next week, for real this time

DIRAC projects

DIRAC:

Issues by milestone:

v8.0:
- We added a workaround for ARC CEs (ARC bug)
- from previous meeting
  - Add config option to prevent some groups from cleaning transformations
    - Federico assigned to me, can I assign to someone from Juno?
  - Don’t block the transformation system if owner is removed
    - Xiaomei to give more details
  - RucioFileCatalogClient.setMetadata returns empty result on success
from previous meeting v9.0:
- Job Pre/Post processing flexibility
  - actually working for this in the CWL prototype
  - Andrei we would actually need to add plugins
v9.1:
- Nothing new

Other issues:

Propagate number of processors from Transformations or Productions down to Jobs description
- Put as reminder, does not seem something we’ll actually solve in DIRAC
Gracefully stopping pilots #8346
- requires some discussion
Get infos from CRIC
- any volunteer?
from previous meeting Proposal for input data resolution format
- also cited in https://github.com/aldbr/dirac-cwl-proto/issues/25
from previous meeting Replacement for BDII2CSAgent #8194
- “nice” discussion
- 17th July
  - recent answers in the GGUS ticket above from AP seems to point in the right direction
  - nevertheless, Federico is trying to grab info on if we can do a “CEs” crawler ourselves. ARC CEs seem to provide the necessary information, issues are from HTCondorCEs
- This has now been discussed in the GridPP project management board. Conclusion was unfortunately “input from GOCDB team needed”
- 09 Oct 2025 GridPP ops team asked UK sites to implement Alessandro’s suggestion. The following issues were observed:
  - It’s not documented (beyond a soon-to-be-archived broadcast)
  - Free form encourages typos and misremembered VO names (vo.na62.gridpp.ac.uk vs na62.vo.gridpp.ac.uk) – this would not happen with predefined scopes
  - RAL as the Tier 1 hit the length limit for extension as it supports too many VOs
  - Sigh.
    - Last message from Federico:
    For those of you who are running a HTCondor CE, would you mind investigating if the content of auth-map + accounting-map files could be made public? Maybe we are lucky and it’s a trivial thing to be done.
    - Daniela This has gone to the HTCondor developers via RAL, but so far no answer.
- 13th Nov New updates in https://helpdesk.ggus.eu/#ticket/zoom/3427

PRs discussed:

fix: use cachedmethod instead of cached for methods #8361
- why still a draft? should then be applied also to the scitag PR
[9.0] feat: SSHCE export inputs/import outputs as a JSON file to handle more jobs in parallel #8362
- draft but hotfixed in LHCb
[9.0] Avoid endless loop in MonitoringReporter #8363
- just a question to answer in there
feat: verify CS config commit using diracx model validation #8351
- This is ready (?)
[9.0] Subprocess: use psutil for killing processes #8359
- This is ready – needs careful review
[9.0] Use DIRACX_CA_PATH to communicate to diracx and S3 #8364
from previous meeting PoolCE and RAM (issue raised in https://github.com/DIRACGrid/DIRAC/issues/7853#issuecomment-2948565279): https://github.com/DIRACGrid/DIRAC/pull/8232
- Federico made progresses, ready to be reviewed since last week. The logic in there shoud be checked
- Andrei matching done with tags, which is not ideal – there is also a limit on the number of tags that we can use.
- cburr should really use MB instead of GB
from previous meeting add scitag support
- “re-written” and ready to merge

WebApp:

Created v6.0.0 version

Pilot:

PR for removing Python2 support re-created (previous one introduced a bug)
- will be merged into master after a DIRAC(X) hackathon
- from previous meeting if you really want to keep py2 support, there’a tag
PR for using the PreferredURL pattern
- ready (finally also tests are all passing)

DIRACOS:

Documentation:

from previous meeting tasks: https://github.com/search?q=org%3ADIRACGrid is%3Aissue state%3Aopen label%3Adocumentation &type=issues
- including some for diracx-charts and diracx-web
Andrei Need documentation on how releases are made

management

Having again issues for /cvmfs/dirac.egi.eu (the syncing was not happening, ticketed, etc.). No monitoring on their side, urgent need to move out.
from previous meeting new /cvmfs/dirac.cern.ch repository created – CERN ticket
- action on cburr to populate it (using LHCb “machinery”)

DB12

NTR
from previous meeting Federico “I asked Igor if he wanted to become one if not the main maintainer”
- revived https://github.com/DIRACGrid/DB12/issues/9
- create https://github.com/DIRACGrid/DB12/issues/15 and made a suggestion that can be accepted
- PR https://github.com/DIRACGrid/DB12/pull/16 created – will wait for Alexandre for review
from previous meeting We should agree on a strategy on how to do things here, as PRs can’t just be merged:
- there are reports created that depend from it, merging/releasing on a random Tuesday is not the way to go
- can we have parallel benchmarks? not clear actually if that’s doable
- Federico maybe create a “2026” release?
  - we should see if there would be time for that

Rucio

from previous meeting CTAO submitted draft PR https://github.com/DIRACGrid/DIRAC/pull/8327 and issue https://github.com/DIRACGrid/dirac/issues/8331
- 23 Oct 25 Federico I asked Cedric to review these and other Rucio issues/PRs, but “no time”.

Tests

DiracX:

dependabot alert for web
- N interesting TR

Issues and Discussions

CS repo directory problems in container deployment #700
- How to use LocalGitConfigSource?
- Chris and Chris will answer on this ASAP
There are several “Good first issues” looking for volunteers

PRs discussed:

Renovate version maintenance for Autorest client generation #699
feat (Configuration): add documentation to CS attributes; sub_from_preferred_username uses cache #696
- - 2 other PRs for generating documentation. 2 of them ready for merge.
from previous meeting Alembic and triggers
- asked Cedric to review it
from previous meeting Add pilot management: create/delete/patch and query #570
- PR looks OK-ish to merge (should not affect existing running code)
from previous meeting feat: deploy gubbins images #527
- Federico I rebased it but still failing

CWL prototype

“moving” with full plan in https://codimd.web.cern.ch/tUL0IUMKT56tsNv3dAf65w?view.
- 8 people working on it, try to use SCRUM

DiracX-charts:

from previous meeting Moved to fixed versions in diracx/values.yaml and added automatic chart version bumping on DiracX releases
- Similarly should be done for diracx-web

DiracX-web:

Next appointments

Meetings:
- Should be DIRAC(X) cert hackathon in 1 week
- BiLD: in 2 weeks
WS/hackathons/conferences:
- DiracX hackathon 21st-22nd January 2026: https://indico.cern.ch/event/1582395/

AOB

It’s again CHEP abstract submission time, deadline 19th December
- Dirac(X) “state of affairs” abstract? Other suggestions?
- Federico will write down an abstract but will not go.
ISIMA forum 20th-21st November (students will be LHCb-hosted). Time to write down project proposals – ideas?
- Federico+ChrisB+Alexandre writing down a proposal for a new Jobs Matching mechanism for DiracX: https://codimd.web.cern.ch/eOU4kV3jRTSSJq43FXvhwg#
  - idea would be that student prepares a prototype for stress tests (e.g. 10M jobs in queue, how fast can we match?)
  - (new) description of job requirements to be used will need your suggestion
- Any non-LHCb have a proposal and want to be a non-official supervisor?
from previous meeting DIRAC as an “HSF affiliated project” : https://hepsoftwarefoundation.org/projects/affiliated.html
- 13th Nov 2025 no news, but in the meantime this was presented at few forums (…). In the meantime Ruslan restarted the evaluation.

LHCbDIRAC

from previous meeting The CS in lhcbdiracx has not been updated lately because of new fields added which do not pass the validation
- so no new users in the last 10 days
- protection from that is in the works (issue in DIRAC attached to Natthan)
from previous meeting The CS overloading is still on-the-edge, needs Pilots to respect the PreferredURL regex
*
Several MRs in the system, reminders to update the existing ones and check what’s actually ready for merge
Bookkeeping
- We should change the passwords of our dirac users - new should be at least 15 characters long
- from previous meeting
  - First priority is the repartition, that will happen either the last week of October or the first of November. After that, the existing PRs will be evaluated
- add materialized view for processing paths optimization
  - this should be merged first and the next ones use it
  - Alexey it does not work for me, see comment in MR
- reworked addProcessing
- reworked getproductionporcpassname
LHCb abstracts for CHEP go in https://codimd.web.cern.ch/4mOg3RgsT1eVTnb4KT3Tuw?edit