RAL Tier1 Experiments Liaison Meeting

Wednesday 21 May 2025, 13:30 → 15:50 Europe/London

Access Grid (RAL R89)

13:30 Experiment Operational Issues

1 ATLAS Operations Report

Speakers: Brij Kishor Jashal (Rutherford appelton laboratory), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

ATLAS liason meeting notes Brij

2 CMS Operations Report

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

Errors yesterday are on CMS side.

A couple of warnings on squid - 

1. many database queries from the site have connected directly to the Frontier servers or backup proxies, with a high rate of queries not going through the local squid(s): T1_UK_RAL\tcmscernct\t255759\t3922096
2. No IPv6 access

A small spike of job failures yesterday. Otherwise good performance.

3 LHCb Operations Report

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

• Raw data distribution starts very soon (planned for today or tomorrow)
  • 1PB free on antares LHCb allocation (which is 2024-25 FY), so need update it soonish.
• ECHO problems (faulty disk on the 30th of April causing some slow IOPs + slow IOPs for uknown reason on the 7th of May) caused file loss and corruption (GGUS 683184)
  • Search for corrupted files is finished
    • A handful of files found, the ones that have other replicase have already been re-replicated to RAL
    • That (hopefully) concludes the incident
• There were a lot of upload failures from HLTFarm and NIPNE to RAL
  • Not our fault
    • Network issues at NIPNE and HLTFarm connectivity problems

 

CVMFS (sorry, misplaced):

• There is a proposal to make RAL Stratum-1 an official mirror of the EESSI repository. Any thoughts on that?
• squid0[56] machines were added to the cvmfs-squid alias. That caused some problems for ATLAS and CMS
  • PTR records were added to reverse DNS zone as well, e.g.

    $ host 130.246.183.211
    211.183.246.130.in-addr.arpa domain name pointer cvmfs-squid.gridpp.rl.ac.uk.
    211.183.246.130.in-addr.arpa domain name pointer squid06.gridpp.rl.ac.uk.
    211.183.246.130.in-addr.arpa domain name pointer cms-squid.gridpp.rl.ac.uk.
    211.183.246.130.in-addr.arpa domain name pointer atlas-squid.gridpp.rl.ac.uk.
    
    

    That's most probably the cause of the issues. As of this morning, the DNS change (full, e.g. for both Forward and Reverse zones) is rolled-back.

4 ALICE Operations Report

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

5 LSST Operations Report

Speakers: Mathew Sims, Timothy Noble (Science and Technology Facilities Council STFC (GB))

• File Movement causing disruption to jobs at RAL for the DC2 dataset
  • Meeting today to hopefully sort the RC2 dataset (to supersede DC2) should be taking around 1 month 
• Message stack in place and monitroing for it is evolvong well
  • Once complete and documented plans to share with other sites
•  

14:00 Tier-1 Projects

6 Anatares Upgrade

New EOS nodes
Repack Progress

Speakers: George Patargias, Thomas Byrne

7 Echo Upgrade

Dell 24 Storage Deployment
New gateways
SSD Storage

Speakers: Robert Appleyard, Thomas Byrne

8 Varnish For ATLAS

Speaker: Brij Kishor Jashal (Rutherford appelton laboratory)

9 XRootD Development

Speakers: Alexander Rogovskiy (Rutherford Appleton Laboratory), Jyothish Thomas (STFC)

10 Utilizing GPUs

Speakers: Jyoti Prakash Biswal (Rutherford Appleton Laboratory), Thomas Birkett

14:45 AOB

11 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore

Weekly Report 21 May 2025.docx

Weekly Report 21 May 2025.pdf

12 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore