RAL Tier1 Experiments Liaison Meeting

Wednesday 18 Oct 2023, 13:30 → 15:00 Europe/London

Access Grid (RAL R89)

14:00 → 14:01 Major Incidents Changes

14:01 → 14:02 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB)), Kieran Howlett (STFC RAL)

Weekly Report 18 October 2023.docx

Weekly Report 18 October 2023.pdf

14:02 → 14:03 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

14:04 → 14:05 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

14:05 → 14:06 Experiment Operational Issues

14:15 → 14:16 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

I hear the OPN connection is cut and needs repair. CMS jobs and transfers appear unaffected, except perhaps a brief drop in efficiency,  hard to say if it's a direct effect - maybe not.

Because of the above issue, i will not run my test from CERN to Echo yet (I mentioned this yesterday on Slack). This should test the OPN, and see what throughput we can achieve. This is part of a series of tests with all CMS T1s.

No significant SAM test failures since last Wednesday. This is presumably due to new Echo gateways being installed (on the new network) which are alleviating the gateway pressures. 

CMS are running up to the cap of 12k cores, with generally good efficiency. Some spikes in failure rate to investigate - file access errors.

CMS are doing tests with the ARM farm at Glasgow, following completion of ATLAS tests.

14:16 → 14:17 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

• GGUS-163297:
  • RAL-LCG2: deletion errors; solved.
  • The deletion errors are not currently present, and the volume of data stored is back below the pledge -- observation from past two weeks.
  • We also have more gateways.

 

• Rucio servers were unreachable over IPv6 between 14:20 and 16:00 CEST on 16 October: 
  • CERN-wide networking issue OTG0145014.
  • Blacklisting event due to stage-out timeout. Recovered.
  • Some UKI sites were affected, not RAL. 

 

• ATLAS jobs seem fine with the gateway upgrade. 

 

• Followed up the email thread on SWAP issues at RAL.
  • Seems several other sites have it. 

 

• RAL-LCG2_MCORE brokeroff is set to retire. 
  • Prod/MCORE PQ used to help scheduling, which was unhappy at following gshare when switching S-MCORE and back. Assume scheduling better now.
  • Partitioned resources bad for latency and following gshare.

14:20 → 14:21 VO Liaison LHCb

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

jobs.png

transfers_from.png

transfers_to.png

LHCb:

• 15 files were lost on ECHO
  • Investigation is ongoing

ALICE:

• VObox was damaged yesterday, OK now.

14:25 → 14:28 VO Liaison LSST

Speaker: Timothy John Noble (Science and Technology Facilities Council STFC (GB))

14:30 → 14:31 VO Liaison Others

14:31 → 14:32 AOB

14:32 → 14:33 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))