RAL Tier1 Experiments Liaison Meeting

Wednesday 29 Jul 2026, 13:30 → 15:50 Europe/London

Access Grid (RAL R89)

13:30 → 13:34 Site Operations

13:34 → 13:35 Experiment Operational Issues

13:35 → 13:40 ATLAS Operations Report

Speakers: Brij Kishor Jashal (Rutherford Appleton Laboratory), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

Brij ATLAS liason meeting notes of the week

13:40 → 13:45 CMS Operations Report

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

SAM tests:

• On Friday yet another instance of all the ARC-CE token tests not running - applies to all sites using ARC-CE. Reported it again. 
• On Monday Alex reported CEs were out of free space and Tom investigated. This caused a few hours of failures for CMS SAM tests until it was fixed.

 

Production:

• A large number of job failures over the weekend - but all User Analysis, has now subsided. 
• Note, a small number of jobs are currently reading data from offsite. The amount of data read has increased and the read time has also jumped - to be monitored.

 

AAA machinery: 

• SVC42 in prod; one more machine to come shortly. Requested SAM tests for these.
• Currently doing a test to see how the XCache performs - other AAA gateways have been temporarily turned off.
• Possibility of doing more formalised testing in future when XRootD load-testing tool is ready.

 

/unmerged:

• Deletion on-going (crashed in screen previously)

 

File dump file is being produced again. There was a problem with the cron job which stopped it. CMS are trying to get this back into usage for the Consistency Checking.

13:45 → 13:50 LHCb Operations Report

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

Operational issues:

• RAL CEs ran out of space on Sunday evening [GGUS:1003424]
  
  • Due to a huge CRL produced by GEANT RSA 1 CA
  • Resulted in submission issues and job failures for LHCb
  • Fixed
• Problems accessing files on ECHO from RAL WNs due to xrootd crashes [GGUS:1003341]
  • The fix is being deployed to the prod farm
• Problematic Simulation jobs at RAL and other sites
  • Improving slowly (old MCReconstruction productions are being processed)
• Minor upload failures from RAL WNs to ECHO yesterday and today due to new xrootd image propagation

13:50 → 13:55 ALICE Operations Report

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

Any news on mlsensor installations (GSTSM-756)?

13:55 → 14:00 LSST Operations Report

Speakers: Thomas Birkett, Timothy Noble (Science and Technology Facilities Council STFC (GB))

• Request to move from using HTTPS to DAVS at RAL
  • Moved over to davs as requested - but caused widespread failures
    • Investigating this is because a MOVE has been implemented for other sites to have more reliable data staging
    • But MOVE is not supported by Echo / Object Stores
    • Jyothish has put in a PR to attempt to fix their code base to not try and use a MOVE at RAL 
      • One person commended - will talk to Jyothish specifically
    • Another suggestion is that we should server side replace the MOVE with a COPY and DELETE -
      • Jyothish thinks this will take a number of iterations to get right, and also slow down RAL server operations
  • Now have only one repo using davs to test and show any differences once RPs are accepted, others are using HTTPS for now so the site is not down for LSST
• Jobs running OK when not using DAVS:
  
• No Data flowed to RAL due to no data being produced by Rubin Observatory due to the storm
  Last update I saw was 21st that they thought the storm was passing / calming - but roads and bridges to the site have been destroyed so if repairs are needed it may take some time

14:00 → 14:01 Tier-1 Projects

14:05 → 14:10 XRootD Development

Speakers: Alexander Rogovskiy (Rutherford Appleton Laboratory), Jyothish Thomas (STFC)

kernel patching reboots happening, 
xrootd 6 with streamed checksums to be deployed this week on external gateways
this was deployed on the batch farm already.

14:15 → 14:20 Utilizing GPUs

Speakers: Dr Brij Kishor Jashal (Rutherford Appleton Laboratory), Thomas Birkett

14:25 → 14:45 Hardware deployment

Echo SN
WN
GPU

Speaker: Jacob Ward

14:45 → 14:46 AOB

14:50 → 14:59 Summary of Operational Status and Issues

Speakers: Brian Davies (Science and Technology Facilities Council STFC (GB)), Darren Moore, Thomas Birkett

Weekly Report 29 July 2026.docx

Weekly Report 29 July 2026.pdf

15:00 → 15:05 Any other Business

Speakers: Brian Davies (Science and Technology Facilities Council STFC (GB)), Darren Moore