RAL Tier1 Experiments Liaison Meeting

Wednesday 20 Sept 2023, 13:30 → 15:00 Europe/London

Access Grid (RAL R89)

14:00 → 14:01 Major Incidents Changes

14:01 → 14:02 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB)), Kieran Howlett (STFC RAL)

Weekly Report 20 September 2023.docx

Weekly Report 20 September 2023.pdf

14:02 → 14:03 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

14:04 → 14:05 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

14:05 → 14:06 Experiment Operational Issues

14:15 → 14:16 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

Much better this week...SAM tests have been mostly green since the move of WN output traffic from external to (as yet unavailable on the new network) "internal" gateways. This WN output is mostly coming from ATLAS and LHCb since CMS are still using gsiftp gateways for WN upload.

However, FTS traffic was heavy yesterday, and the external gateways suffered a bit...SAM tests gave us a failing day accordingly.

Also last Wednesday I took CMS out of drain regardless of the gateway issue. I felt CMS jobs could still achieve good performance, and that was the case until now.

DI made a change this morning to try to put 'new' gateways on the new network with IPv6...but we are not sure it is working as expected.

 

Schedule as of Monday:

• LHC very high beta* run in progress; Heavy-ION afterwards/next week; end-of-year-technical-stop starts Oct 30th, 2023

14:16 → 14:17 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

DAOD_test_steps.pdf

• RAL-LCG2 staging Issue “File not found” (https://ggus.eu/index.php?mode=ticket_info&ticket_id=162827 ); on hold:
  • All Antares Tier-1 EOS nodes were upgraded to EOS 4.8.105 on 14 September.
  • The situation is being monitored. 

 

• RAL-LCG2: deletion errors (https://ggus.eu/index.php?mode=ticket_info&ticket_id=163297); in progress:
  • Updated the ticket on 19 September from ATLAS perspective => much improved situation. 
  • Additional improvements to the gateways to be implemented soon.

 

• ATLAS jobs failing with: "failed to close file descriptor: bad file descriptor": https://stfc.atlassian.net/browse/GS-131 
  • The situation was monitored, and the fix was found to be satisfactory.
  • Ticket closed. 

 

• Investigations into why the batch farm is forcing ATLAS into swap memory.
  • Spotted by Tom Birkett on 18 September.
  • e.g., https://bigpanda.cern.ch/job/5966764388/

 

• Lack of ATLAS jobs recently?

 

• DAOD-on-demand demo test: 
  • https://docs.google.com/document/d/118ryBh628w7hSsHPUOjrKvVrnJsm4H_L0DjasvtRlR8/edit
  • The detailed steps are in the attachment. 
  • Went ahead on 18 September as expected.
    • Containers for the DAOD sample to be used in the DAOD-on-demand tape test at RAL were created -- user.xzhao:data.DAOD.for_RAL_tape_test_00; user.xzhao:mc.DAOD.for_RAL_tape_test_00.
    • Rucio rules to transfer to RAL tape RSEs were also created: 91a058326e19410abcfe4aea76adf020; 249b323ad2bb4db883260a62277a2e43.
    • DDM Ops approval on its way. 

14:20 → 14:21 VO Liaison LHCb

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

14:25 → 14:28 VO Liaison LSST

Speaker: Timothy John Noble (Science and Technology Facilities Council STFC (GB))

14:30 → 14:31 VO Liaison Others

14:31 → 14:32 AOB

14:32 → 14:33 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))