RAL Tier1 Experiments Liaison Meeting

Wednesday 6 Sept 2023, 12:30 → 14:30 Europe/London

Access Grid (RAL R89)

13:00 → 13:01 Major Incidents Changes

13:01 → 13:02 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB)), Kieran Howlett (STFC RAL)

Weekly Report 06 September 2023.docx

Weekly Report 06 September 2023.pdf

13:02 → 13:03 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

13:04 → 13:05 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

13:05 → 13:06 Experiment Operational Issues

13:15 → 13:16 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

13:16 → 13:17 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

RAL_WNs.pdf

• RAL-LCG2 staging Issue “File not found” (https://ggus.eu/index.php?mode=ticket_info&ticket_id=162827 ); On hold:
  • EOS 4.8.104 has been built (to fix the WebDAV issue).
  • Upgradation of antares preprod instance to the EOS version by George P. 
    • Endpoints: root://antares-preprod.stfc.ac.uk:1094; https://antares-preprod.stfc.ac.uk:9000
    • EOS ATLAS directory: /eos/antarespreprodtier1/atlas/
  • Also tested (thanks to George and Alex R.):
    • Error when trying copy an unstaged file via https => the bug is fixed!
    • Files can't be copied back via https => a new bug?
      • George has already posted about it in the EOS forum.
  • The next update on the ticket is likely within this week. 

 

• ATLAS jobs failing with: "failed to close file descriptor: bad file descriptor": https://stfc.atlassian.net/browse/GS-131 
  • There does not seem to be a need to create a GGUS ticket, as the issue is local.
  • Thanks to Tom Birkett's fix, there has not been any bad file descriptor issue for quite some time now (last checked on Monday).
  • The fix seems to work in a very positive way so far. 
  • Hope to discuss on the JIRA if there is a new development. 

 

• RAL continues to counter the stage-out errors on a daily basis => HammerCloud exclusions. 
  • E.g.,  http://bigpanda.cern.ch/job?pandaid=5954540873
      ==> modificationHost: slot1_11@lcg2303.gridpp.rl.ac.uk  pilot:::1152 File transfer timed out during stage-out: hc_test:output.1.20741172-6bde-46d0-a998-a02093ae81de_48996.pool.root to RAL-LCG2-ECHO_SCRATCHDISK, copy command timed out: TimeoutException: Timeout reached, timeout=312 seconds')]:failed to transfer files using copytools=['rucio'].
  • Any suggestions?

 

• RAL sees high Panda job failures during the past 24 hours (see the attachment).

 

• The Echo Gateways look to be in a really bad state - why don't we have the new ones available yet ? 

 

• DAOD-on-demand demo test:
  • https://docs.google.com/document/d/118ryBh628w7hSsHPUOjrKvVrnJsm4H_L0DjasvtRlR8/edit
  • Xin Zhao (BNL):
    • "If RAL is ready, we can start the tests at RAL, first step would be to write the sample DAOD 
      datasets to RAL tape. I will create the rucio rules for tape write (and for tape read as well later).
      What we will need from sites will be to clean up tape disk buffer before tape read test, help 
      monitor internal tape situations (e.g. how busy is the staging queue with other production 
      requests etc) and help investigate any site related issues, if any. "

13:20 → 13:21 VO Liaison LHCb

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

slides.pdf

Operational issues:

1. Failed transfers to ECHO (gateway issues?)
2. Farm restarted yesterday, all LHCb jobs were lost.
3. There is a lack of productions from LHCb, that's why the number of running jobs is low.

13:25 → 13:28 VO Liaison LSST

Speaker: Timothy John Noble (Science and Technology Facilities Council STFC (GB))

13:30 → 13:31 VO Liaison Others

13:31 → 13:32 AOB

13:32 → 13:33 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))