RAL Tier1 Experiments Liaison Meeting

Wednesday 30 Apr 2025, 13:30 → 15:50 Europe/London

Access Grid (RAL R89)

13:30 Experiment Operational Issues

1 ATLAS Operations Report

Speakers: Brij Kishor Jashal (Rutherford appelton laboratory), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

ATLAS liason meeting notes Brij

2 CMS Operations Report

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

Red days for SAM on Tuesday/Wednesday due to the continuation of network issues/interventions reported last week. 

Observed large discrepancy between CMS (monit) and RAL (Vande) monitoring on running cores. This turned out to be scheduling inefficiency on CMS side with slow ramp up of scheduling agents at FNAL. 

Tape downtime caused CMS to go into drain several times. The Rucio status (for Echo) was overridden by Data Management. Katy overrode the status to keep jobs running. CMS needs to treat this better and not send sites into drain just because tape is down.

A few spikes in job failures and low efficiency which may be related to network blips (long read times).

RAL FTS removed entirely from CMS-Rucio operations.

3 LHCb Operations Report

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

jobs.png

transfers_from.png

transfers_to.png

Issues:

• Network problems on the 23rd of April
  • Affected LHCb jobs
  • Looks OK since the morning of the 23rd
• LHCb drained on Saturday
  • Lack of jobs

4 ALICE Operations Report

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

5 LSST Operations Report

Speakers: Mathew Sims, Timothy Noble (Science and Technology Facilities Council STFC (GB))

• DC2 pipeline complete, but working with CM team to standardise the data return process
  • The script to return data is hard-coded in places, which all rely on URI, which is currently incorrect
    • Working on moving data to the correct location now
    • Approx 4 million files 
    • writing a script to use multiprocessing with Python to move data
• Will run DC2 again with the weekly code from week 18 from Friday (1st May)
  • And will continue to run every other week going forwards to assist with site and code base testing and troubleshooting
• After DC2 data return process complete will work on getting the RC2 data registered in a butler repository for smaller test but with pre-curser data
• LSST want to use a UK FTS for European transfers, so will want to either use SKA FTS or LCGFTS - or at least use it for fail over
  • Not using a FTS for a time and wanting to use it for failover could lead to further failures
  • Do ATLAS and CMS use a single central FTS for all transfers or do they also use geographically closest ones?
    • From talking to other devs Rucio currently uses only the FTS related to the destination 

14:00 Tier-1 Projects

6 Anatares Upgrade

New EOS nodes
Tape Robotics downtime

Speakers: George Patargias, Thomas Byrne

7 XRootD Development

Speakers: Alexander Rogovskiy (Rutherford Appleton Laboratory), Jyothish Thomas (STFC)

8 Varnish For ATLAS

Speaker: Brij Kishor Jashal (Rutherford appelton laboratory)

14:45 AOB

9 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore

Weekly Report 30 April 2025.docx

Weekly Report 30 April 2025.pdf

10 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore