RAL Tier1 Experiments Liaison Meeting

Wednesday 2 Aug 2023, 12:30 → 14:30 Europe/London

Access Grid (RAL R89)

13:00 → 13:01 Major Incidents Changes

13:01 → 13:02 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB)), Kieran Howlett (STFC RAL)

Weekly Report 02 August 2023.docx

Weekly Report 02 August 2023.pdf

13:02 → 13:03 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

13:04 → 13:05 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

13:05 → 13:06 Experiment Operational Issues

13:15 → 13:16 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

CMS went into drain on Saturday after SAM tests failing on Wednesday, Friday and Saturday. Katy brought CMS back into production on Monday as tests had been green on Sunday and up until that point on Monday. Still the occasional failure and warning state indicating gateways are not in a 'perfect' state, and this is supported by the gw tests on Vande.  We had a discussion yesterday about how to better monitor gws within the redirector, and better alert the on-call.

The source of the errors was webdav and root tests, with slow-reads a typical failure. Possible cause is the current number of gateways - we are missing two of the newer ones due to the networking changes, although we have gw8 in place to try to mitigate. For this reason, Katy has not yet moved WN stage-out from gsiftp to davs. 

CMS now has a cap of 8k, although after coming back from drain, CMS did not have sufficient job pressure across the world to exploit the 8k cap. No sign of saturation on any external links. At time of writing we are back to 8k cores.

Still want to move to Tape REST API, but waiting to discuss with Alastair. Also, ATLAS are seeing problems with davs for Antares, see their minutes. 

 

13:16 → 13:17 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

• GGUS tickets:
  • RAL-LCG2 staging Issue “File not found” (https://ggus.eu/index.php?mode=ticket_info&ticket_id=162827) -- since 24 July 2023.
    • The issue is being discussed in https://its.cern.ch/jira/browse/EOS-5771  (from https://cern.service-now.com/service-portal?id=ticket&table=incident&n=INC3537619).
    • A problem where a WebDav transfer of a non-staged files reports as successful, but results in a zero-byte (or no output file) being created.
  • RAL-LCG2_MCORE has failing jobs due to connection problems (https://ggus.eu/index.php?mode=ticket_info&ticket_id=162909) -- since 28 July 2023.
    • While percentage of job failures caused by stage-in timeouts is not very high, the number of failures is not insignificant (hundreds per day) e.g., https://bigpanda.cern.ch/job?pandaid=5919722792

 

• ATLAS jobs failing with: "failed to close file descriptor: bad file descriptor": https://stfc.atlassian.net/browse/GS-131

 

 

13:20 → 13:21 VO Liaison LHCb

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

slides.pdf

Tickets:

• Redirector deployment
  • LHCb switched to redirector for https transfers
  • No improvement in upload efficiency
• RAL -> SARA network issue
  • Asymmetric routing problems, under investigation
• Vector read
  • Closed!
  • But there is still some room for improvement

Operational issues

• LHCb had stopped exporting XrdSecGSISRVNAMES after the switch to apptainer
  • Authentication was broken for certain versions of xrootd as a result
  • Fixed by LHCb
  • We should note that all env variables are now cleared by default, we should ask them to add exceptions should we need them
• A lot of failed uploads due to gateway issues
• Insufficient number of productions from LHCb

13:25 → 13:28 VO Liaison LSST

Speaker: Timothy John Noble (Science and Technology Facilities Council STFC (GB))

13:30 → 13:31 VO Liaison Others

13:31 → 13:32 AOB

13:32 → 13:33 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))