BiLD – 04/09/2025

At CERN: Federico, Christophe, Christopher, Henryk
On Zoom: Alexandre, Simon, Jorge, Hideki, Alexei
Apologies:

Previous meetings

Last BiLD was several weeks ago, in July

DIRAC communities roundtable

LHCb:

Federico+Christophe+Christopher+Alexandre+Vladimir

Belle2

Cedric, Hideki, Ueda, Dhiraj

Trying to update one of the dev server to v9
- CS conversion not passing the pydantic

CLIC

Andrè

IHEP

Xiaomei

from previous meeting Started using chatbot for DIRAC – using llamaindex
- 17th July ready to combine the work, student will try to push to Alexandre’s repository (not yet done)

EGI:

Andrei

from previous meeting Started work on the agent to send “energy/CO2” records (from DIRAC, but not only) to the GreenDIGIT MetricsDB service. Working together with Henryk.

CTA:

Natthan

GridPP:

Daniela, Simon, Janusz

Topics from GitHub discussions and bots

only un-answered DIRAC and DiracX topics with discussion updates:
- NTR

Releases

DIRAC

v8.0.76
- published on 15th July, nothing since then
v9
- Few tags being made for integration DIRACCommon (see below)
- Will tag v9.0 today or tomorrow

diracx

v0.0.1a4x
*

Release planning, tests and certification

Certification machines
- Will make sure that is “restored” before the workshop
Next hackathon(s)
- Maybe at the DIRAC workshop, on the first morning.

DIRAC projects

DIRAC:

Issues by milestone:

v8.0:
- NTR
v9.0:
- BaseJobDescriptionModel uses Operations inappropriately closed by creation of DIRACCommon (see below)
v9.1:
- NTR Federico the biggest problem is that they are all assigned to me

Other issues:

from previous meeting Replacement for BDII2CSAgent #8194
- “nice” discussion
- Daniela is going to check if (as she suspects) GOCDB is actually underwritten by GridPP. Maybe (can’t promise) “finances” might be an angle.
- 17th July
  - recent answers in the GGUS ticket above from AP seems to point in the right direction
  - nevertheless, Federico is trying to grab info on if we can do a “CEs” crawler ourselves. ARC CEs seem to provide the necessary information, issues are from HTCondorCEs
    - Last message from Federico:
    For those of you who are running a HTCondor CE, would you mind investigating if the content of auth-map + accounting-map files could be made public? Maybe we are lucky and it’s a trivial thing to be done.
    - Daniela will check the above

PRs discussed:

[8.0] IdP clients per user group
- just one added question there
DIRACCommon (few PRs):
- new package created and deployed to pypi: https://pypi.org/project/DIRACCommon/
- lives inside DIRAC code and has the same versioning
- DiracX will depend from it (and not anymore from DIRAC)
- all the stateless functionalities of DIRAC are moved there (e.g. not using gLogger, gConfig, Operations, etc.)
- DIRACCommon will be released (with the same tag) every time DIRAC is tagged
from previous meeting add scitag support
- draft PR waiting for Christophe review. More comments in it
- several VOs (including LHCb, Belle2, ilc) can already provide scitags (recognized with the APIs)
- 04/09 approved informally by Marian Babik
- Christophe can re-use the “activity”, no need for new DB field
from previous meeting PoolCE and RAM (issue raised in https://github.com/DIRACGrid/DIRAC/issues/7853#issuecomment-2948565279): https://github.com/DIRACGrid/DIRAC/pull/8232
- 17th July PR updated
- 4th Sept completing it at the workshop’s hackathon

WebApp:

Pilot:

from previous meeting Pilot migration
- diracx pilot route
from previous meeting PR feat: Adding JWT support alongside X509 auth
- the new Pilot command can call directly the route, no need to use the CLI
- the integration tests for this will be set up once diracx is updated with the connected diracx PR

DIRACOS:

Documentation:

from previous meeting web.diracgrid.org
- for the moment deployed at gitlab at Lyon. Will be moved to standard DIRACGrid github repo at some point
- the suggestion is to do it already
- Andrei is it OK to deploy (the landing pages) on github pages?
- charts.diracgrid.org used for diracx chart hosting: https://github.com/DIRACGrid/diracx-charts/settings/pages
from previous meeting tasks: https://github.com/DIRACGrid/DIRACx/issues?q=is%3Aissue state%3Aopen label%3Adocumentation

management

from previous meeting new /cvmfs/dirac.cern.ch repository created – CERN ticket
- action on @cburr to populate it (using LHCb “machinery”)

DB12

Rucio

Tests

DiracX:

dependabot alert for web
- NTR

Issues

from previous meeting Federico wrote down an “epic” with 5 sub-tasks: https://github.com/DIRACGrid/DIRACx/issues/562 for the DiracX accounting/monitoring after consulting with CERN experts
- 17th July We gave Ewoud (CERN IT) a task to try this one out on a 6-months popularity data (from LHCb). Current stoppers:
  - Grafana does not yet support plotting of OpenSearch rolled-up data (seems to be OK with data from ElasticSearch though, so probably something that will eventually be sorted out)
    - ticket open by CERN MONIT team to Grafana
  - 4th Sept More worringly, rollup jobs do not seem to work on DataStreams (work only on standard indices)

PRs discussed:

fix: jobparameters and jobattributes pydantic models
from previous meeting Alembic and triggers
- asked Cedric to review it
from previous meeting Add pilot management: create/delete/patch and query #570
- PR looks OK-ish to merge (should not affect existing running code)
from previous meeting feat: deploy gubbins images #527
- waiting for https://github.com/DIRACGrid/diracx-charts/pull/158 to be approved and merged
  - 17th July [MERGED]

DiracX-charts:

Moved to fixed versions in diracx/values.yaml and added automatic chart version bumping on DiracX releases
- Similarly should be done for diracx-web

DiracX-web:

from previous meeting Theau implemented a way for share app states (dump in JSON, reload through web)
- follow-up issue: https://github.com/DIRACGrid/diracx-web/issues/348
- 5th June the new proposal is to have a DB where to stored snippets for the state. Basically, like the UserProfileDB in DIRAC.
New SearchBar for the JobMonitor
New SunBurst diagram
Several minor fixes
- Few PRs to merge

Next appointments

Meetings:
- BiLD: October 9th
WS/hackathons/conferences:
- DIRAC Users’ Workshop
  
  📅 17–20 September 2025 | 📍 IHEP, Beijing, China
  🔗 indico.cern.ch/e/duw11
  * Connect with the DIRAC community
  * Meet the developers
  * Discover DiracX
  * Share your insights
  The workshop will feature talks from users, administrators, and developers — plus a hands-on DiracX Hackathon.
  
  Poster in https://cernbox.cern.ch/remote.php/dav/public-files/1egpqkseV0GvqlO/DIRAC_Poster.pdf – make use of it!

AOB

from previous meeting DIRAC as an “HSF affiliated project” : https://hepsoftwarefoundation.org/projects/affiliated.html
- 4th Sept Review “assigned”

LHCbDIRAC

Bookkeeping MRs:
- add materialized view for processing paths optimization
  - this should be merged first and the next ones use it
  - Alexey it does not work for me, see comment in MR
- reworked addProcessing
- reworked getproductionporcpassname
- 4th Sept No progress on any of the above – probably no progress before October
from previous meeting StorageReport app available in
- https://lhcbdiracx-cert.app.cern.ch/
- few minor issues still to be sorted out before accepting the PR
- 26th June pending some code re-organization
New Bookkeeping browser web app on lhcbdiracx-web
- Quite some progress made, but not ready to go public. Beta-testers will be welcome at some point but not yet
- Little-to-nothing in common with the current LHCbWebApp app
Bookkeeping DB work:
- Very promising results from partitioning tests. It might be even done without need for DB locking