RAL Tier1 Experiments Liaison Meeting

Wednesday 11 Jun 2025, 13:30 → 15:50 Europe/London

Access Grid (RAL R89)

13:30 → 13:31 Experiment Operational Issues

13:35 → 13:40 ATLAS Operations Report

Speakers: Brij Kishor Jashal (Rutherford appelton laboratory), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

ATLAS liason meeting notes Brij

13:40 → 13:45 CMS Operations Report

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

Screenshot 2025-06-11 at 13.51.48.png

Screenshot 2025-06-11 at 13.52.06.png

Another update on previously reported SAM test issues:

1. svc20 (AAA server) not showing problems since being added with new memory limits. 
2. Network problems - a few hiccups last week but not significant for CMS. Problem hopefully fixed on Friday/today. 
3. 'Connection' test for Antares endpoints in warning due to no IPv6 - how are the tests for the new EOS nodes going? 

Another good week in production for CMS, with lots of cores and good performance. 

Transfers:

From last week, the large number of file exists errors likely due to network disruption - automated clean up mechanism was very effective and errors went to zero without Katy having to do anything.

Errors with Echo as source mentioned last week were mostly transfers failing to Antares as mentioned above. 

I observe that this week we are running a lot of data Processing jobs, and much of the data is likely sourced from CERN. However, at a glance, jobs reading Offsite are performing better than those reading Onsite! N.B. Update - those jobs failing with Onsite reads are almost entirely User Analysis.

13:45 → 13:50 LHCb Operations Report

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

5d2040e3c6af56f85e3d8a550a6765f9.png

a14f3fa12c6e9a41740447454eb0a03a(1).png

News:

• LHCb drained last Sunday due to a problem with one of the LHCbDIRAC VMs
  • Recovered by Monday morning

Issues:

• Network outages [GGUS ticket]:
  • All looks ~OK since Friday afternoon
  • The issue was fully fixed today, so we can close the ticket?
• ceph-svc24 was intermittently crashing yesterday
  • Due to a bug in the new checksum dump code
    • Fixed now
  • That caused a (minor) increase in upload failures.
• Spike of deletion failures this morning
  • All failures were due to timeouts.

CVMFS:
Stratum-1 servers were rebooted yesterday. That stopped snapshots due to removal of the /run/cvmfs.local directory. The directory was created manually yesterday evening, that enabled snapshots again. A proper fix is being prepared.

 

 

13:50 → 13:55 ALICE Operations Report

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

13:55 → 14:00 LSST Operations Report

Speakers: Mathew Sims, Timothy Noble (Science and Technology Facilities Council STFC (GB))

• RC2 data movement getting there, with some blockers on the US site, these have been overcome and data now mostly registered in Rucio - and working on the rest

 

• IngestD version increase to v20 today by Mat

 

• DC2 w18 now complete with only a minor issue that was resolved and reported
  • Job requested 4GB and used around 14/15GB 
    • Didn't fail at Lancs or IN2P3 due to not having the same sort of limits set (3x to request then kill)
    • This has been reported back to CM team
  • Will now run w22

 

 

 

14:00 → 14:01 Tier-1 Projects

14:15 → 14:25 Anatares Upgrade

New EOS nodes
Repack Progress

Speakers: George Patargias, Thomas Byrne

14:25 → 14:35 XRootD Development

Speakers: Alexander Rogovskiy (Rutherford Appleton Laboratory), Jyothish Thomas (STFC)

14:35 → 14:55 Echo SN24 Install

14:55 → 15:05 Utilizing GPUs

Speakers: Jyoti Prakash Biswal (Rutherford Appleton Laboratory), Thomas Birkett

15:05 → 15:25 SWIFT-HEP SSD storage

15:25 → 15:26 AOB

15:26 → 15:35 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore

Weekly Report 11 June 2025.docx

Weekly Report 11 June 2025.pdf

15:35 → 15:40 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore