RAL Tier1 Experiments Liaison Meeting

Wednesday 5 Aug 2026, 13:30 → 15:50 Europe/London

Access Grid (RAL R89)

1 Site Operations

13:34 Experiment Operational Issues

2 ATLAS Operations Report

Speakers: Brij Kishor Jashal (Rutherford Appleton Laboratory), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

Brij ATLAS liason meeting notes of the week

3 CMS Operations Report

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

SAM tests:

• On Thursday all CMS sites went into drain due to nobody noticing the HC jobs cease running a few days previously. This was tracked to a config file that CERN had given additional security protections. Fixed within the day. 

 

Production:

• A drop in running cores at the moment, below pledge, but assume this is simply fair share balancing (CMS had higher running cores earlier in the week). 
• CMS-Rucio also suffered from the oversized CRL issue seen at RAL last week. 

 

AAA machinery: 

• Both recycled machines SVC42 and 41 now in prod; SAM tests are green. 
• Other AAA gateways have been turned off for now.
• New XRootD load-testing tool used to test performance.
• Regular crashes observed  (42) which didn't impact production it seemed - Jyothish fixed the config to accommodate the core dump size and we observe no crashes since (except one, but no core dump visible). 

 

AAA tests, with the load-test tool pushing on top of normal production activity:

• Memory: Climbed a bit, not as bad as old machines, but worth keeping an eye on
• Throughput: High, almost filling the NIC at one point (25Gb NIC). 
• Affect on Echo: None observed (Jyothish can comment)

 

/unmerged:

• Deletion still on-going, parallelisation may not have scaled well, so we may have to ensure this is running for the next 1-2 months. 

 

DC27 progress: Katy put the numbers for CMS desired rates into her all-VO spreadsheet, ready for other VOs to start adding info too.

FTS4: ATLAS are testing in Production, agreed with Steve Murray to have CMS test also in ~3 weeks assuming no show-stoppers

4 LHCb Operations Report

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

News:

• Staging campaign started on Monday
  • Antares and ECHO are doing well

Operational issues:

• Many upload failures from RAL WNs to ECHO GGUS:1003466
  • WN gateways seem overloaded, a lot of slow ops can be seen in the logs
  • Gateways recalculating the checksums from scratch for every file, due to mtime check misconfiguration
    • That's probably the reason behind the overloads
    • Fix has been deployed to the preprod farm
• High failure rate for Simulation jobs
  • It was ~OK last week (MCReconstruction fix working), though some MCSimulation jobs are still affected
    • LHCb people are working on it
• Harica CRL problem (GGUS:1003424) fixed
• Xrootd crashing due to long filenames (GGUS:1003341) fixed 
• Proxy update failed on CERN ETF machine last Thursday
  • As a result ETF tests were failing for all sites for a few hours
  • Not our fault, obviously

5 ALICE Operations Report

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

6 LSST Operations Report

Speakers: Thomas Birkett, Timothy Noble (Science and Technology Facilities Council STFC (GB))

14:00 Tier-1 Projects

7 XRootD Development

Speakers: Alexander Rogovskiy (Rutherford Appleton Laboratory), Jyothish Thomas (STFC)

xcache has been running on AAA all week 

• production has been transparent and getting up to 22Gbps with no impact on Echo IOPs
  
• silently crashing ~5 times per day... no cause detected yet and no crashes today after increasing the dump size yesterday

kernel patching on gateways

alice token plugin in compilation for xrootd 6

 

8 Utilizing GPUs

Speakers: Dr Brij Kishor Jashal (Rutherford Appleton Laboratory), Thomas Birkett

9 Hardware deployment

Echo SN
WN
GPU

Speaker: Jacob Ward

20260805_Ral-Tier1-Experiments-Liaison-Meeting_Commissioning-Update.pdf

14:45 AOB

10 Summary of Operational Status and Issues

Speakers: Brian Davies (Science and Technology Facilities Council STFC (GB)), Darren Moore, Thomas Birkett

Weekly Report 05 August 2026.docx

Weekly Report 05 August 2026.pdf

11 Any other Business

Speakers: Brian Davies (Science and Technology Facilities Council STFC (GB)), Darren Moore