BiLD-Dev

→ Europe/Zurich
2/R-014 (CERN)

2/R-014

CERN

10
Show room on map
Federico Stagni (CERN)
Description

Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.

Zoom: BiLD
https://cern.zoom.us/j/62504856418?pwd=TU1kb01SOFFpSDBJeWVBdU9qemVXQT09

Meeting ID: 62504856418
Passcode: 12345678
 

Zoom Meeting ID
62504856418
Host
Federico Stagni
Useful links
Join via phone
Zoom URL
 
 

BiLD – 14/11/2024

At CERN: Federico, André, Christophe, Alexandre, Christopher
On Zoom: Simon, Hideki, Andrei, Alexey, Xiaomei, Janusz, Igor, Jorge, Cedric
Apologies: Daniela


Follow-up from previous meetings

  • Last BiLD was October 31st
  • Last DIRAC certification hackathon on November 7th

DIRAC communities roundtable

LHCb:

Federico+Alexandre+Christopher+Vladimir

  • Running over 300k jobs in parallel (new record!)

  • fromPreviousMeeting

    • Stressing also the WMS, some jobs running twice (same jobID picked up by 2 different pilots)
      • Federico will re-check
      • 14th Nov issue solved some time ago

ILC/Calice/FCC

André

  • NTR

EGI

Andrei

  • Few sites moved to recent version of HTCondor, so no more X509 proxies and campaign needed. Need to use tokens or SSL
    • not using IAM, but CheckIN (which is multi-VO, and a special entitlement is needed). A plugin is there for HTCondor and ARC, but sites need to be configured to install this script
    • All the VOs and all the sites need to be done at the same time
  • fromPreviousMeeting
    • Setting up the diracx test system with K3S in parallel with DIRAC 9.0. Struggling with setting up the basic setup (cert-manager, DEX, …)

Belle2

Hideki, Cedric

  • Asked questions about OpenSearch index management

GridPP:

Daniela, Simon, Janusz

  • fromPreviousMeeting
    • We would like to upgrade to the latest v8 version, but we have too many problems with unrenewed proxies. Not every VO went down the WLCG route of 7 day proxies, so we tend to see them more often: https://github.com/DIRACGrid/DIRAC/discussions/7842
      • Andrei the current CEs are not renewing the pilot proxy, so at the moment the only way is to have long pilot proxies
        • Alexandre ARC is doing it (upgrading the delegation), I do not think HTCondorCE does it. Anyway this would not help with the bundled proxy that we are using now
        • proxy renewing itself? does not seem “correct”
        • Andrei proxy delegation thread in SiteDirector?
        • or maybe a Dirac-only internal token/proxy (without VOMS) only for renewing the user proxy
          • maybe re-using the same solution used for the CloudCE. Simon will have a look
  • Solution to the above: https://github.com/DIRACGrid/DIRAC/pull/7886
    • looks correct – merged!

NICA

Igor

  • Issues with DB12. Culprit maybe proxmox on debian, and for the moment just exclude these nodes
    • Suggestion in the ticket to move to wall time
    • Federico to give suggestions on where to put this

Topics from GitHub/Discussions

only un-answered DIRAC and DiracX topics with discussion updates:


DIRAC releases

  • v8r0
    • v8.0.57 + v8.0.58
    • TransformationSystem
      FIX: (#7861) getTasksToSubmit consider tasks inserted by 30 seconds or more

DIRAC projects

DIRAC:

Issues by milestone:

Other issues:

PRs discussed:

WebApp:

  • NTR

Pilot:

  • from previous meeting Janusz some doc to write

DIRACOS:

Documentation:

  • NTR

OAuth2:

  • NTR

management

diraccfg

  • NTR

DB12

  • fromPreviousMeeting Igor made studies

Rucio

  • NTR

Tests


DiracX:

Issues

  • Assigned issue for OpenTelemetry to Jorge
    • started looking into it, but Grafana from the demo does not work

PRs discussed:

  • PR(s) for using PostGres: we can not use it as of now, so unfortunaly we will close, but the work has been useful. Christophe will create an issue for not forgetting what we will want to do in the future
  • https://github.com/DIRACGrid/diracx/pull/269
    • Chris I will check offline
  • Little to no movement otherwise

DiracX-charts:

DiracX-web:

  • Lots of “bumps” PRs: they will be closed once the below is merged
  • feat: extension management
  • fromPreviousMeeting
    • Alexandre
      • Improvements to JobMonitoring app
      • Added the user’s details at login
      • Some tests for the extension needs to be added
    • asked Ryunosuke to be a reviewer for the DiracX-web PRs

Release planning, tests and certification

  • Certification machines
    • dteam IAM setup more difficult than what was planned

      • one client created by Federico seemed “cursed”, ended up just creating a new one
        • seems like once created it, we should not modify it
      • created also a client for pilots submission
        • Federico contacted a few sites for allowing it (NCBJ, IN2P3, CERN)
          • NCBJ allows “all” client IDs, and indeed it works
      • we should document which options are needed for creating these clients
    • Federico will allow a LegacyAdaptor for JobMonitoring (it was there in the past, why not anymore?)

DiracX is crashing:

2024-11-14 08:25:40,952 - INFO:     10.76.163.2:38662 - "GET /api/docs/ HTTP/1.1" 307 Temporary Redirect
2024-11-14 08:25:40,952 - INFO:     10.76.163.2:38650 - "GET /api/docs/ HTTP/1.1" 307 Temporary Redirect
2024-11-14 08:25:40,954 - INFO:     10.76.163.2:38664 - "GET /api/docs HTTP/1.1" 200 OK
2024-11-14 08:25:40,954 - INFO:     10.76.163.2:38670 - "GET /api/docs HTTP/1.1" 200 OK
2024-11-14 08:25:50,951 - INFO:     10.76.163.2:45892 - "GET /api/docs/ HTTP/1.1" 307 Temporary Redirect
2024-11-14 08:25:50,952 - INFO:     10.76.163.2:45880 - "GET /api/docs/ HTTP/1.1" 307 Temporary Redirect
2024-11-14 08:25:50,954 - INFO:     10.76.163.2:45902 - "GET /api/docs HTTP/1.1" 200 OK
2024-11-14 08:25:50,954 - INFO:     10.76.163.2:45916 - "GET /api/docs HTTP/1.1" 200 OK
2024-11-14 08:25:52,390 - INFO:     2001:1458:d00:14::16f:0 - "GET /api/auth/legacy-exchange?preferred_username=fstagni&scope=vo%3Adteam+group%3Adteam_pilot+property%3AGenericPilot+property%3ALimitedDelegation&expires_minutes=255157 HTTP/1.1" 500 Internal Server Error
2024-11-14 08:25:52,390 - ERROR:    Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/opt/conda/lib/python3.11/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
  |     yield
  |   File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/base.py", line 186, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/opt/conda/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 763, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/opt/conda/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
    |     result = await app(  # type: ignore[func-returns-value]
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    |     return await self.app(scope, receive, send)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    |     await super().__call__(scope, receive, send)
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
    |     raise exc
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
    |     await self.app(scope, receive, _send)
    |   File "/opt/conda/lib/python3.11/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 579, in __call__
    |     await self.app(scope, otel_receive, otel_send)
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    |     await self.app(scope, receive, send)
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/base.py", line 185, in __call__
    |     with collapse_excgroups():
    |   File "/opt/conda/lib/python3.11/contextlib.py", line 158, in __exit__
    |     self.gen.throw(typ, value, traceback)
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    |     raise exc
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/base.py", line 187, in __call__
    |     response = await self.dispatch_func(request, call_next)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/diracx/routers/__init__.py", line 475, in dispatch
    |     response = await call_next(request)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/base.py", line 163, in call_next
    |     raise app_exc
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/base.py", line 149, in coro
    |     await self.app(scope, receive_or_disconnect, send_no_error)
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     raise exc
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
    |     await route.handle(scope, receive, send)
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
    |     await self.app(scope, receive, send)
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
    |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     raise exc
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
    |     response = await f(request)
    |                ^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/fastapi/routing.py", line 291, in app
    |     solved_result = await solve_dependencies(
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/fastapi/dependencies/utils.py", line 640, in solve_dependencies
    |     solved = await run_in_threadpool(call, **solved_result.values)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
    |     return await anyio.to_thread.run_sync(func, *args)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    |     return await get_async_backend().run_sync_in_worker_thread(
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2441, in run_sync_in_worker_thread
    |     return await future
    |            ^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 943, in run
    |     result = context.run(func, *args)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/diracx/core/config/__init__.py", line 105, in read_config
    |     hexsha, modified = self.latest_revision()
    |                        ^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/diracx/core/config/__init__.py", line 215, in latest_revision
    |     self._pull()
    |   File "/opt/conda/lib/python3.11/site-packages/cachetools/__init__.py", line 814, in wrapper
    |     v = method(self, *args, **kwargs)
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/diracx/core/config/__init__.py", line 212, in _pull
    |     self.repo.remotes.origin.pull()
    |   File "/opt/conda/lib/python3.11/site-packages/git/remote.py", line 1120, in pull
    |     proc = self.repo.git.pull(
    |            ^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/git/cmd.py", line 986, in <lambda>
    |     return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
    |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/git/cmd.py", line 1598, in _call_process
    |     return self.execute(call, **exec_kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/site-packages/git/cmd.py", line 1262, in execute
    |     proc = safer_popen(
    |            ^^^^^^^^^^^^
    |   File "/opt/conda/lib/python3.11/subprocess.py", line 1026, in __init__
    |     self._execute_child(args, executable, preexec_fn, close_fds,
    |   File "/opt/conda/lib/python3.11/subprocess.py", line 1885, in _execute_child
    |     self.pid = _fork_exec(
    |                ^^^^^^^^^^^
    | BlockingIOError: [Errno 11] Resource temporarily unavailable
    +------------------------------------
  • Next hackathon(s)
    • next week

Next appointments

AOB

  • Projects for ISIMA students
    • Deadline: now!
    • decided to go for 2 projects:
      • LHCb specific (LHCb extension, also for the Web)
      • Pilot and
  • Andrei registered a DIRAC:WMS “support unit” in GGUS

LHCbDIRAC

  • BKK:
    • still on :fire:
      • but denormalisation really improves the performances
        • a column should be added, it can be started to be added even “now”, then code can be (hot)fixed later on
    • repartition to hash of the production ID (hash partition type is a thing in Oracle)
      • this should be done by Kate (it doubles the size…)
  • https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/issues/32#note_8726357 : closed
  • from previous meeting
    • Ryun possibility of running on a subset of a BkQuery for prescaling
      • “Merged”
    • Ryun’s DAG graphs: we can run “DAGs” in productions (sequentially)
      • CWL, one day
  • from previous meeting Some runs without luminosity. Total luminosity is there.
 
There are minutes attached to this event. Show them.
    • 10:00 → 10:10
      Items from Previous BiLD-Dev 10m
    • 10:10 → 10:20
      DIRAC Communities roundtable 10m
    • 10:20 → 10:30
      DIRAC releases 10m
    • 10:30 → 10:55
      DIRAC projects 25m
      • DIRAC
      • WebApp
      • Pilot
      • DIRACOS2
      • VMDIRAC
      • Documentation
      • OAuth2
      • DiracX
      • other externals (include Rucio)
    • 10:55 → 11:00
      Release planning, tests and certification 5m
    • 11:00 → 11:15
      Weekly development(s) focus 15m
    • 11:15 → 11:25
      AOB
      Convener: Federico Stagni (CERN)
    • 11:25 → 11:40
      LHCbDIRAC 15m