RAL Tier1 Experiments Liaison Meeting

Wednesday 13 Sept 2023, 13:30 → 15:00 Europe/London

Access Grid (RAL R89)

14:00 → 14:01 Major Incidents Changes

14:01 → 14:02 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB)), Kieran Howlett (STFC RAL)

Weekly Report 13 September 2023.docx

Weekly Report 13 September 2023.pdf

14:02 → 14:03 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

14:04 → 14:05 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

14:05 → 14:06 Experiment Operational Issues

14:15 → 14:16 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

CMS_last2weeks_13Sept23.pdf

Katy was on holiday last week. SAM tests failed accordingly. 

Webdav and xrootd SAM tests saw a high level of failures - see my slides.

Testing batch farm outputs going to 'new' gateways instead of redirector gateways started at 10am today - this applies to ATLAS and LHCb who were writing back from WNs to Echo with webdav. CMS are still using gsiftp for this.

Redirector also lost connection with one of the two IP addresses on monday - spotted and fixed quickly.

14:16 → 14:17 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

 

• RAL-LCG2 staging Issue “File not found” (https://ggus.eu/index.php?mode=ticket_info&ticket_id=162827 ); On hold:
  • The previous issue with WebDAV reads is resolved. 
  • EOS 4.8.105-1 has been built on CERN repos (discussions: https://eos-community.web.cern.ch/t/new-citrine-release-4-8-104/892/20).
  • Tests ongoing. 

 

• RAL-LCG2: deletion errors (https://ggus.eu/index.php?mode=ticket_info&ticket_id=163297); in progress:
  • Connection timed out, deletion-failed, RAL-LCG2-ECHO_DATADISK.
  • The echo team is aware and looking at it.

 

• Recent Echo transfer issues:
  • Last 7 days: avg 44k HS23.
  • 30% failure rate on jobs (and walltime), ~ 90% of errors are stage-out issues.
  • 150k files queued in FTS for writes to Echo. 

 

• ATLAS jobs failing with: "failed to close file descriptor: bad file descriptor": https://stfc.atlassian.net/browse/GS-131 
  • Change deployed to farm yesterday. 
  • Keeping an eye on the situation.

 

• After the success with OX, SHEF was pointed to QMUL SE yesterday. Things look better now. 

 

• The batch farm DNS change rolled over today at 10:00 AM. 

 

• DAOD-on-demand demo test:
  • https://docs.google.com/document/d/118ryBh628w7hSsHPUOjrKvVrnJsm4H_L0DjasvtRlR8/edit
  • Follow ups:
    • The test doesn't require special setups at sites. Want to test in real production environment, and if needed the data samples will be deleted by DDM in future regular tape deletion campaigns. 
    • In principle, there is no problem and can proceed. We are however currently  undertaking work to reduce the load on the Disk Endpoints (i.e. Echo).  Echo will be used, either via multihop, or for staging tests, it might be better to begin the tests at the start of next week, if that would be OK?

14:20 → 14:21 VO Liaison LHCb

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

Tickets:

• Failed uploads
  • The problem is still present, there are a lot of failed uploads

Operational issues:

• Low number of running LHCb jobs
  • Lack of production requests

14:25 → 14:28 VO Liaison LSST

Speaker: Timothy John Noble (Science and Technology Facilities Council STFC (GB))

14:30 → 14:31 VO Liaison Others

14:31 → 14:32 AOB

14:32 → 14:33 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))