RAL Tier1 Experiments Liaison Meeting

Wednesday 29 Nov 2023, 13:30 → 15:00 Europe/London

Access Grid (RAL R89)

14:00 → 14:01 Major Incidents Changes

14:01 → 14:02 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB)), Kieran Howlett (STFC RAL)

14:02 → 14:03 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

14:04 → 14:05 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

14:05 → 14:06 Experiment Operational Issues

14:15 → 14:16 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

Token migration did not happen. We found that the implementation allowed us to access ATLAS files with a CMS token. Katy to test again today with dev gateways with Jyothish's changes.

Gateways causing some problems this week - SAM tests red for the days Thursday, Friday, Tuesday and likely today as well. Went into drain on Saturday, Katy re-enabled on Sunday.

DC pre-test tomorrow for ATLAS and CMS, transfers from CERN to T1s. Test is for egress from CERN.

Small number of intermittent failures on the CVMFS SAM test. There have been several issues with CVMFS lately. 

Katy to do - put Shoveler monitoring into production (has been in test for probably a year). The collector could potentially be used also for RALPP XRootd traffic monitoring.

The consistency checking at RAL was broken due to a problem with the dump file. Alex and Jyothish and Tom fixed it. The issue was with mismatch of file sizes due to the way the file dump gets written to Echo.

14:16 → 14:17 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

RAL_jobs.jpg

TEST_jobs.jpg

• GGUS-164289:
  • RAL-LCG2: deletion errors; in progress.
  • Being looked after by the CEPH team, no updates yet!
  • Link --> Deletion Error  => the count is high. 

 

• DC24:
  • RAL email needs to be added at the CRIC and the RC (--> Details).
  • lcg-support@gridpp.rl.ac.uk is already there, only atlas-uk-ral-tier1-support@cern.ch is to be added.
  • Mattermost channel.

 

• RAL noticed a dip in jobs during 16:00-05:00 hrs  in the last 24 hours (plot attached).

 

• RAL-LCG2_TEST is now fully set for production jobs.
  • The jobs are looking good, fairly consistent (plot attached).
  • HammerCloud test were failing massively, but seem to be fine today. 

 

• RAL continues to experience the short HammerCloud exclusions. 

 

• To build a new Panda queue for the Pre-Prod batch farm (after things are OK with the TEST queue). 

14:20 → 14:21 VO Liaison LHCb

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

jobs.png

transfers_from.png

transfers_to.png

LHCb:

• Problems with ECHO->Antares transfers
  • Due to missing tape families for 2016 data
• Network issue between RALPP and RAL
• Failed uploads from RAL WN to ECHO
  • Errors are mostly timeouts
• File loss on ECHO
  • recovered (when possible)
• Slow checksum ticket
  • Patch was rolled out to all gateways
  • This has not resolved the issue though
• ETF test does not work for test CE
  • This is because of the lack of python executable in the new OS
• Nano DC tomorrow
  • I'll copy 100 files from CERN to RAL, that should be negligible (it's more for my education) 

ALICE:

• cvmfs issue on WNs last week

14:25 → 14:28 VO Liaison LSST

Speaker: Timothy John Noble (Science and Technology Facilities Council STFC (GB))

14:30 → 14:31 VO Liaison Others

14:31 → 14:32 AOB

14:32 → 14:33 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))