RAL Tier1 Experiments Liaison Meeting

Wednesday 17 Sept 2025, 13:30 → 15:50 Europe/London

Access Grid (RAL R89)

13:44 → 13:45 Experiment Operational Issues

13:45 → 14:05 Site Operations

14:05 → 14:10 ATLAS Operations Report

Speakers: Brij Kishor Jashal (Rutherford Appleton Laboratory), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

ATLAS liason meeting notes Brij

Please find my notes attached in the link. 
I have a clash with another meeting, so most likely won't be able to join. 
but if the other one finishes early, I will join. 
Thanks

14:10 → 14:15 CMS Operations Report

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

Screenshot 2025-09-17 at 11.53.40.png

Screenshot 2025-09-17 at 12.03.56.png

Screenshot 2025-09-17 at 12.05.59.png

Screenshot 2025-09-17 at 13.24.28.png

Screenshot 2025-09-17 at 13.25.00.png

Screenshot 2025-09-17 at 13.27.11.png

Katy was on leave last week. The network incident last Tuesday had little effect on CMS compared to other VOs, as CMS did not drop all jobs, and in fact picked up most of the farm as other VOs drained.

Some SAM test failures for AAA since the network incident. Affects the older machines (gw10/11) but not the newer machine (svc20). On Monday SAM tests were green but gw10/11 did not appear to be serving data. Katy did a restart of services on one machine (gw10). Both machines started serving data (so likely the restarts had no effect). Since last night the service has become more busy (50-100MB/s served, but many thousands of connections). gw10/11 are struggling again with spikes up to 10k connections, bad read amplification and intermittently failing SAM tests. Jyothish put both machines in a sandbox this morning. 

Request for the 2026 tape pledge to be provided early has been granted. I have not yet added the additional capacity as I am working with CMS DM to add some throttling to what Rucio can submit to FTS. This is to protect RAL from too big bursts so that we can continue and complete the repacking to the new hardware tech.

Transfers look good, we hit 13GB/s to Echo from CERN (1hr binning) last week on the 8th.

Job performance generally good, however the slow job read times has not gone away. Looking more carefully, this is hugely dominated by Merge jobs even though the amount of data read by merge jobs has remained consistent and average below 10GB, see images attached. Are merge jobs reading remotely? Seems to be affecting only RAL of the CMS T1s, and having a very negative effect on job efficiency (of course we do not run many of these jobs, so minor effect on overall efficiency). 

 

14:15 → 14:20 LHCb Operations Report

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

Quiet week. Only a few minor issues:

• Lack of MC productions is still present in LHCb
  • Job pressure is unstable as a result, and drains happen frequently
• There were two a small spikes (~200 transfers) of download failures from ECHO to RAL WNs last week
  • Due to xrootd container upgrade on the local gateways

14:20 → 14:25 ALICE Operations Report

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

14:25 → 14:30 LSST Operations Report

Speakers: Mathew Sims, Timothy Noble (Science and Technology Facilities Council STFC (GB))

NTR besides a new graduate has joined the team, Chris Winterburn, who will be taking over responsibility for the messaging stack, Strimzi, etc., from Mathew when they leave in two weeks (ish).

14:30 → 14:31 Tier-1 Projects

14:45 → 14:55 Anatares Upgrade

New EOS nodes
Repack Progress

Speakers: George Patargias, Thomas Byrne

14:55 → 15:05 XRootD Development

Speakers: Alexander Rogovskiy (Rutherford Appleton Laboratory), Jyothish Thomas (STFC)

15:05 → 15:15 Utilizing GPUs

Speakers: Jyoti Prakash Biswal (Rutherford Appleton Laboratory), Thomas Birkett

15:15 → 15:20 SSD Storage Evaluation

15:20 → 15:25 Echo deployment

15:30 → 15:31 AOB

15:31 → 15:40 Summary of Operational Status and Issues

Speakers: Brian Davies (Science and Technology Facilities Council STFC (GB)), Darren Moore

Weekly Report 17 September 2025.docx

Weekly Report 17 September 2025.pdf

15:40 → 15:45 Any other Business

Speakers: Brian Davies (Science and Technology Facilities Council STFC (GB)), Darren Moore