Choose timezone
Your profile timezone:
BiLD-Dev
Description
Bi-Weekly "Loyal" DIRAC developers meeting. And, following, the LHCbDIRAC developers meeting.
Zoom: BiLD
https://cern.zoom.us/j/62504856418?pwd=TU1kb01SOFFpSDBJeWVBdU9qemVXQT09
Meeting ID: 62504856418
Passcode: 12345678
BiLD
BiLD – 15/02/2024
At CERN: Federico, André, Alexandre, Christophe
On Zoom: Christopher, Janusz, Daniela, Simon, Andrei, Hideki, Janusz, Cedric, Ruslan, Vladimir, Alexey
Apologies:
Follow-up from previous meetings
- Last BiLD on Jan 25th
- Last DIRAC certification hackathon 1 week ago: https://trello.com/b/OJhbYH1Y/v900a22
- Still issues with SiteDirectors and Pilots
- SiteDirectors stopped since last Friday
- JobMonitoring now works after installing newest version
- Still issues with SiteDirectors and Pilots
- Last BiLDx two weeks ago
- The 2 weeks deadline for answering https://github.com/DIRACGrid/diracx-web/discussions/63 is today
- Same for the surveys in https://github.com/DIRACGrid/diracx-web/discussions/categories/surveys
DIRAC communities roundtable
LHCb:
Federico+Alexandre+Christophe+Christopher
- Running on latest DIRAC v8 patch, plus hotfixes from [8.0] Support token transfers in FTS in order to run DataChallenge24 (stress-testing)
Belle2
Cedric, Hideki
- DIRAC v8 validation
- Pilot basically working
EE_KEY_TOO_SMALL- André maybe a too small user proxy
ILC/Calice/FCC
André
- Servers: RHEL9, Dirac 8.0.37+HotFixes, pilot Dirac~=8.0
- Issues a few weeks ago because of runit for el9
- Automatically picking up 8.0.37 for pilot not good
- FCC IAM promised for March
EGI
Andrei
-
There was a spike in the user’s activities in recent days that stressed the service: the job rate of more than 200k per day with about 5K running jobs. A lot of small non-parameteric jobs. The stress resulted in timeouts for some service (Accounting, JobStateUpdate) and piling up of failover requests. The following measures were taken:
- Reduced VOMS proxy length requested by RequestTasks, increased VOMS extension time limit - helped to cope with slow reponse of some VOMS servers as more proxies were taken from the cache ;
- Setting several VOs not to require VOMS extension for users (those not using grid storage), only for pilots ;
- Spawning several more instances of affected components (JobStateUpdate, StorageElement, SandboxStore, RequestExecutingAgents) ;
- Upgrading the hosting configuration - more powerful hyperviser hardware, increasing the network quota.
-
After the taken measures, the system is stable at 200K job per day for a couple of days now.
Juno/BES3
Xiaomei
JINR
Igor
- Running on
develpilot and v8.0 from CVMFS:- found bug, fixed in https://github.com/DIRACGrid/Pilot/pull/231 (merged already)
prolink in CVMFS was not updated
GridPP:
Simon, Janusz, Daniela
- Token rollout in production. While the actual submitting of pilot jobs with tokens seems to work, I have not been able to extract any pilot logs from the ARC-CEs.
- The error is either: “Server error while serving getPilotLoggingInfo: setToken() missing 1 required positional argument: ‘valid’”
- Or “Failed to retrieve logging info for https://arcce02.esc.qmul.ac.uk:443/arex/[long arc string]”
- How’s LHCb doing in that regard ?
- Alexandre Same issues, they should be solved now:
- I can get the pilot logs from HTCondorCEs.
- We deployed DIRAC 8.0.36 on Rocky 9 on one of our pre-prod servers. The basics seem to work.
- Federico is the runit RPM what is linked from https://github.com/DIRACGrid/DIRAC/issues/7368 ?
Topics from GitHub/Discussions
only un-answered topics with discussion updates:
DIRAC releases
DIRAC projects
DIRAC:
Issues by milestone:
-
v8.0:
- 10+ open issues, no new ones in the last weeks
- usual reminder for closing/moving old ones
- 10+ open issues, no new ones in the last weeks
-
v9.0:
- 15+ open issues
- Federico solved Add VO to sb_Owners table? and Are Per User Subproxies used?, PRs already merged
- [8.0] Implementation of metadata methods into RucioFileCatalogClient closes issue #7382
- Synchronize IAM to CS from VOMS2CSAgent
- NTR
PRs discussed:
- [8.0] Elastic: index creation only at indexing time
- To avoid indices with 0 documents
- [9.0] remove notification db
- Federico started simplification, ended up removing the whole system
- question: does anyone ever used this DB?
- Will be merged by next BiLD if no objections
- [9.0] feat: split JobWrapper.execute() into 3 submethods
- Only refactoring. First of several PRs
- [9.0] fix: use UTC for plotting
- Patched into the DIRAC certification, working fine
- Solving a looong term issue with DIRAC plots
WebApp:
Pilot:
- Merge of
develtomaster: nope!- actually, found an issue, already fixed.
- Janusz some doc to write
DIRACOS:
- Made a new release with
libxml2downgraded. Issue opened togfal2for proper fix
Documentation:
- NTR
OAuth2:
- Published CVE https://github.com/DIRACGrid/DIRAC/security/advisories/GHSA-59qj-jcjv-662j
- All sys admins were already notified in advance. Patch already included
management
- Always upload releases to CVMFS
- from previous meeting
- Also the upload of pilot files to CVMFS could be done same way
- Right now the cron-job runs hourly
- Few issues discovered (the releases were not properly propagated – CVMFS issue at RAL)
- from Stratum0 to Stratum1
- fixed by hand from time to time
- https://cvmfs-monitor-frontend.web.cern.ch/ might be needed
- Andrei found how to publish also the ARM architecture. Still done by hand.
- from previous meeting
diraccfg
- NTR
DB12
Alexandre
- from previous meeting
- Ewoud opened PR for py3.11 (in progress)
- missing the Intel vs AMD
- Federico Can we have DB12 run on ARM
- André : I access two “real” ARM machines through OpenStack that can be used for the tests
- Federico: passed to Ewoud
- hepscore and ARM: https://ggus.eu/index.php?mode=ticket_info&ticket_id=164939 (Imperial’s ARM is still very much in the “setting it up stage”
- Ewoud opened PR for py3.11 (in progress)
Rucio
- NTR
Tests
- Federico Will create a repo for running “nightly” system tests targeting the DIRAC cert setup (sorry, no time these weeks)
Release planning, tests and certification
-
Certification machines
- lbcertifdirac70 machine:
- we should move to EL9 box in Q1 2024 (CC7 EOL June 2024)
- As said, this should be done outside of CERN
- Volunteers?
- 2 options:
- completely new setup (new DBs). Federico I would prefer this one
- keep the current CERN DBs (MySQL, OpenSearch), + Grafana.
- As said, this should be done outside of CERN
- we should move to EL9 box in Q1 2024 (CC7 EOL June 2024)
- lbcertifdirac70 machine:
-
Next hackathon(s)
- in 2 weeks
AOB
Next BiLDX: Feb 22nd
Next hackathon: Feb 29th (!)
Next BiLD: Mar 7th
-
Next DIRAC Users’ workshop in Lyon https://indico.cern.ch/event/1341205 June 19th
-
Next DIRAC+X hackathon: https://indico.cern.ch/event/1376672/
- 9-10 April
- as usual, do register to participate
LHCbDIRAC
- Vladimir will create/deploy the (patch) releases from now on
- v11.0: deploy board in https://trello.com/b/Ep0PAkbv/deploy-110
- NTR
- Bookkeping MRs:
- Federico I will install it in LHCbDIRAC certification setup as soon as possible
- DiracX ? – can be disabled
- The tables could be added already
- Also something in the Web should be updated
- https://gitlab.cern.ch/lhcb-dirac/LHCbDIRAC/-/issues/24
- We definitely do not need data quality per file
- Federico I will install it in LHCbDIRAC certification setup as soon as possible
- VOMS backend will be shutdown in June 24
- we need to make sure that all links will be updated
- Christophe will do adiabatically over the coming weeks
- OpenSearch migration
- Not yet
- Alma9
- CC7 EOL June 24
- 2 machines (lbvobox900 and lbvobox901) created by Joel
- https://github.com/DIRACGrid/DIRAC/issues/7368
- Andrei access to Glasgow ARM farm?
- Federico asked Dave Britton, no answer, should ask again. At the same time we can use LHCb Core SW machine for DIRAC tests
- Chris B Any objection in removing the production request creation from the LHCb WebApp?
- Seems no objections
- First just remove the button. Cleanup later on.
There are minutes attached to this event.
Show them.