BiLD-Dev
Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.
Zoom: BiLD
https://cern.zoom.us/j/62504856418?pwd=TU1kb01SOFFpSDBJeWVBdU9qemVXQT09
Meeting ID: 62504856418
Passcode: 12345678
BiLD – 14/11/2024
At CERN: Federico, André, Christophe, Alexandre, Christopher
On Zoom: Simon, Hideki, Andrei, Alexey, Xiaomei, Janusz, Igor, Jorge, Cedric
Apologies: Daniela
Follow-up from previous meetings
- Last BiLD was October 31st
- Last DIRAC certification hackathon on November 7th
- Board: https://github.com/orgs/DIRACGrid/projects/19
- enabled
dteamVO, most of time spent understanding what was wrong - again, did not go really far
- only a few people joined!
- enabled
- Board: https://github.com/orgs/DIRACGrid/projects/19
DIRAC communities roundtable
LHCb:
Federico+Alexandre+Christopher+Vladimir
-
Running over 300k jobs in parallel (new record!)

-
fromPreviousMeeting
- Stressing also the WMS, some jobs running twice (same jobID picked up by 2 different pilots)
- Federico will re-check
- 14th Nov issue solved some time ago
- Stressing also the WMS, some jobs running twice (same jobID picked up by 2 different pilots)
ILC/Calice/FCC
André
- NTR
EGI
Andrei
- Few sites moved to recent version of HTCondor, so no more X509 proxies and campaign needed. Need to use tokens or SSL
- not using IAM, but CheckIN (which is multi-VO, and a special entitlement is needed). A plugin is there for HTCondor and ARC, but sites need to be configured to install this script
- All the VOs and all the sites need to be done at the same time
- fromPreviousMeeting
- Setting up the diracx test system with K3S in parallel with DIRAC 9.0. Struggling with setting up the basic setup (cert-manager, DEX, …)
Belle2
Hideki, Cedric
- Asked questions about OpenSearch index management
GridPP:
Daniela, Simon, Janusz
- fromPreviousMeeting
- We would like to upgrade to the latest v8 version, but we have too many problems with unrenewed proxies. Not every VO went down the WLCG route of 7 day proxies, so we tend to see them more often: https://github.com/DIRACGrid/DIRAC/discussions/7842
- Andrei the current CEs are not renewing the pilot proxy, so at the moment the only way is to have long pilot proxies
- Alexandre ARC is doing it (upgrading the delegation), I do not think HTCondorCE does it. Anyway this would not help with the bundled proxy that we are using now
- proxy renewing itself? does not seem “correct”
- Andrei proxy delegation thread in SiteDirector?
- or maybe a Dirac-only internal token/proxy (without VOMS) only for renewing the user proxy
- maybe re-using the same solution used for the CloudCE. Simon will have a look
- Andrei the current CEs are not renewing the pilot proxy, so at the moment the only way is to have long pilot proxies
- We would like to upgrade to the latest v8 version, but we have too many problems with unrenewed proxies. Not every VO went down the WLCG route of 7 day proxies, so we tend to see them more often: https://github.com/DIRACGrid/DIRAC/discussions/7842
- Solution to the above: https://github.com/DIRACGrid/DIRAC/pull/7886
- looks correct – merged!
NICA
Igor
- Issues with DB12. Culprit maybe proxmox on debian, and for the moment just exclude these nodes
- Suggestion in the ticket to move to wall time
- Federico to give suggestions on where to put this
Topics from GitHub/Discussions
only un-answered DIRAC and DiracX topics with discussion updates:
- Is it possible to make Transformation System aware of Tag
- answered in the discussion, but remarked that this IMHO is not a distributed computing issue
- fromPreviousMeeting How to install TokenManager
- Andrei can backport PR [9.0] DISET version of the TokenManager service
- 14th Nov done during the meeting
- Andrei can backport PR [9.0] DISET version of the TokenManager service
DIRAC releases
- v8r0
- v8.0.57 + v8.0.58
- TransformationSystem
FIX: (#7861) getTasksToSubmit consider tasks inserted by 30 seconds or more
DIRAC projects
DIRAC:
Issues by milestone:
- v8.0:
- fromPreviousMeeting job status transition in ReqClient vs JobStatus
- Andrei will answer
- Removal of “Systems” part of CS: PR merged, now we need the update script (which needs to be tested): https://github.com/DIRACGrid/DIRAC/issues/7808
- fromPreviousMeeting job status transition in ReqClient vs JobStatus
- v9.0:
- NTR
- non-cvmfs version of pilot does not run at RAL-LCG2 (Issue #7657)
- from previous meeting Pilot status for pilots in PollTime related sleep cycle
- interesting discussion, feel free to contribute
PRs discussed:
- [8.0] Use plain proxy for pilot bundle
- merged
- [8.0] the TransformationCleaningAgent forces jobs to be killed
- for cleaning productions.
- [8.0] adding possibility to bulk insert in JobLoggingDB
- specifically for parametric jobs
WebApp:
- NTR
Pilot:
- from previous meeting Janusz some doc to write
DIRACOS:
- PR there for updating ancient version of
stomp- already hotfixed in LHCb instance
- from previous meeting Installation errors for diracos
Documentation:
- NTR
OAuth2:
- NTR
management
- from previous meeting Always upload releases to CVMFS
- maybe fixed? Need to check again with the next release, stopping the cron job
diraccfg
- NTR
DB12
- fromPreviousMeeting Igor made studies
Rucio
- NTR
Tests
DiracX:
- Federico Should we move https://gitlab.cern.ch/chaen/cs_store_dirac_certif to github?
- has to be a private repo
Issues
- Assigned issue for OpenTelemetry to Jorge
- started looking into it, but Grafana from the demo does not work
PRs discussed:
- PR(s) for using PostGres: we can not use it as of now, so unfortunaly we will close, but the work has been useful. Christophe will create an issue for not forgetting what we will want to do in the future
- https://github.com/DIRACGrid/diracx/pull/269
- Chris I will check offline
- Little to no movement otherwise
DiracX-charts:
- feat(diracx-web): support npm workspaces
release-pleaseaction not fully supported, creating issues
DiracX-web:
- Lots of “bumps” PRs: they will be closed once the below is merged
- feat: extension management
- fromPreviousMeeting
- Alexandre
- Improvements to JobMonitoring app
- Added the user’s details at login
- Some tests for the extension needs to be added
- asked Ryunosuke to be a reviewer for the DiracX-web PRs
- Alexandre
Release planning, tests and certification
- Certification machines
-
dteamIAM setup more difficult than what was planned- one client created by Federico seemed “cursed”, ended up just creating a new one
- seems like once created it, we should not modify it
- created also a client for pilots submission
- Federico contacted a few sites for allowing it (NCBJ, IN2P3, CERN)
- NCBJ allows “all” client IDs, and indeed it works
- Federico contacted a few sites for allowing it (NCBJ, IN2P3, CERN)
- we should document which options are needed for creating these clients
- one client created by Federico seemed “cursed”, ended up just creating a new one
-
Federico will allow a
LegacyAdaptorfor JobMonitoring (it was there in the past, why not anymore?)
-
DiracX is crashing:
2024-11-14 08:25:40,952 - INFO: 10.76.163.2:38662 - "GET /api/docs/ HTTP/1.1" 307 Temporary Redirect
2024-11-14 08:25:40,952 - INFO: 10.76.163.2:38650 - "GET /api/docs/ HTTP/1.1" 307 Temporary Redirect
2024-11-14 08:25:40,954 - INFO: 10.76.163.2:38664 - "GET /api/docs HTTP/1.1" 200 OK
2024-11-14 08:25:40,954 - INFO: 10.76.163.2:38670 - "GET /api/docs HTTP/1.1" 200 OK
2024-11-14 08:25:50,951 - INFO: 10.76.163.2:45892 - "GET /api/docs/ HTTP/1.1" 307 Temporary Redirect
2024-11-14 08:25:50,952 - INFO: 10.76.163.2:45880 - "GET /api/docs/ HTTP/1.1" 307 Temporary Redirect
2024-11-14 08:25:50,954 - INFO: 10.76.163.2:45902 - "GET /api/docs HTTP/1.1" 200 OK
2024-11-14 08:25:50,954 - INFO: 10.76.163.2:45916 - "GET /api/docs HTTP/1.1" 200 OK
2024-11-14 08:25:52,390 - INFO: 2001:1458:d00:14::16f:0 - "GET /api/auth/legacy-exchange?preferred_username=fstagni&scope=vo%3Adteam+group%3Adteam_pilot+property%3AGenericPilot+property%3ALimitedDelegation&expires_minutes=255157 HTTP/1.1" 500 Internal Server Error
2024-11-14 08:25:52,390 - ERROR: Exception in ASGI application
+ Exception Group Traceback (most recent call last):
| File "/opt/conda/lib/python3.11/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
| yield
| File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/base.py", line 186, in __call__
| async with anyio.create_task_group() as task_group:
| File "/opt/conda/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 763, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/opt/conda/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
| await super().__call__(scope, receive, send)
| File "/opt/conda/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
| raise exc
| File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
| await self.app(scope, receive, _send)
| File "/opt/conda/lib/python3.11/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 579, in __call__
| await self.app(scope, otel_receive, otel_send)
| File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
| await self.app(scope, receive, send)
| File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/base.py", line 185, in __call__
| with collapse_excgroups():
| File "/opt/conda/lib/python3.11/contextlib.py", line 158, in __exit__
| self.gen.throw(typ, value, traceback)
| File "/opt/conda/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/base.py", line 187, in __call__
| response = await self.dispatch_func(request, call_next)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/diracx/routers/__init__.py", line 475, in dispatch
| response = await call_next(request)
| ^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/base.py", line 163, in call_next
| raise app_exc
| File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/base.py", line 149, in coro
| await self.app(scope, receive_or_disconnect, send_no_error)
| File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/opt/conda/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/opt/conda/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
| await route.handle(scope, receive, send)
| File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
| await self.app(scope, receive, send)
| File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/opt/conda/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/opt/conda/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
| response = await f(request)
| ^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/fastapi/routing.py", line 291, in app
| solved_result = await solve_dependencies(
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/fastapi/dependencies/utils.py", line 640, in solve_dependencies
| solved = await run_in_threadpool(call, **solved_result.values)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
| return await anyio.to_thread.run_sync(func, *args)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
| return await get_async_backend().run_sync_in_worker_thread(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2441, in run_sync_in_worker_thread
| return await future
| ^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 943, in run
| result = context.run(func, *args)
| ^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/diracx/core/config/__init__.py", line 105, in read_config
| hexsha, modified = self.latest_revision()
| ^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/diracx/core/config/__init__.py", line 215, in latest_revision
| self._pull()
| File "/opt/conda/lib/python3.11/site-packages/cachetools/__init__.py", line 814, in wrapper
| v = method(self, *args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/diracx/core/config/__init__.py", line 212, in _pull
| self.repo.remotes.origin.pull()
| File "/opt/conda/lib/python3.11/site-packages/git/remote.py", line 1120, in pull
| proc = self.repo.git.pull(
| ^^^^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/git/cmd.py", line 986, in <lambda>
| return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/git/cmd.py", line 1598, in _call_process
| return self.execute(call, **exec_kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/site-packages/git/cmd.py", line 1262, in execute
| proc = safer_popen(
| ^^^^^^^^^^^^
| File "/opt/conda/lib/python3.11/subprocess.py", line 1026, in __init__
| self._execute_child(args, executable, preexec_fn, close_fds,
| File "/opt/conda/lib/python3.11/subprocess.py", line 1885, in _execute_child
| self.pid = _fork_exec(
| ^^^^^^^^^^^
| BlockingIOError: [Errno 11] Resource temporarily unavailable
+------------------------------------
- Next hackathon(s)
- next week
Next appointments
-
Meetings:
- hackathon: next week
- BiLD: in 2 weeks
-
WS/hackathons/conferences:
- DiracX hackathon: 14th-15th January 2025, CERN https://indico.cern.ch/event/1458873/
- Dirac&Rucio mini-workshop and hackathon: 16th-17th January 2025, CERN https://indico.cern.ch/event/1443765/
- Dirac Users’ Workshop: 17th-20th September 2025 - https://indico.cern.ch/e/duw11
AOB
- Projects for ISIMA students
- Deadline: now!
- decided to go for 2 projects:
- LHCb specific (LHCb extension, also for the Web)
- Pilot and
- Andrei registered a DIRAC:WMS “support unit” in GGUS
LHCbDIRAC
- BKK:
- still on
- but denormalisation really improves the performances
- a column should be added, it can be started to be added even “now”, then code can be (hot)fixed later on
- but denormalisation really improves the performances
- repartition to hash of the production ID (hash partition type is a thing in Oracle)
- this should be done by Kate (it doubles the size…)
- still on
- https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/issues/32#note_8726357 : closed
- from previous meeting
- Ryun possibility of running on a subset of a BkQuery for prescaling
- “Merged”
- Ryun’s DAG graphs: we can run “DAGs” in productions (sequentially)
- CWL, one day
- Ryun possibility of running on a subset of a BkQuery for prescaling
- from previous meeting Some runs without luminosity. Total luminosity is there.