RAL Tier1 Experiments Liaison Meeting

Wednesday 23 Aug 2023, 12:30 → 14:30 Europe/London

Access Grid (RAL R89)

13:00 → 13:01 Major Incidents Changes

13:01 → 13:02 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB)), Kieran Howlett (STFC RAL)

Weekly Report 23 August 2023.docx

Weekly Report 23 August 2023.pdf

13:02 → 13:03 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

13:04 → 13:05 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

13:05 → 13:06 Experiment Operational Issues

13:15 → 13:16 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

Significant webdav failures on Sunday and Monday gave 2 red days in SAM tests causing CMS to go into drain on Tuesday. However things improved by lunchtime on Monday after Jyothish did gateway restarts on 3 gateways, so Katy brought CMS back into production soon after number of running cores started to drop. 

Echo crashes ongoing.

Current cap is 12k cores. Performance in general looks good other than some spikes of failures explained by problems described below. Network usage spiked to near saturation on the OPN link connecting WNs to the outside world - likely caused by CMS remote reads. Network usage is now back to low levels, e.g. a couple of 10Gb/s.

Problems on batch farm starting on Friday - zombie containers seemingly caused by Fermilab VO access to particular CVMFS repo. Differences seen between monit and Vande. Containers being locked up. Farm had to drain out and all nodes had to be rebooted at the start of the week. CMS generally kept hold of its WNs (except during brief drain) which is typical of the long pilot time of CMS. But still there was a time when only two thirds of CMS jobs were doing anything - however strangely the job time efficiency in Vande remained above 80%. I do not understand. 

CMS saw a spike in WN stage-out errors at the start of the above problems. Tom Birkett said the disks were 'locked' for I/O. The error for CMS jobs was that the output file on the WN was unavailable, which is consistent with Tom's comment. 

CMS still using gridftp for WN stage-out. Not affected by separate stage-out issue seen by ATLAS and LHCb.

13:16 → 13:17 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

• RAL-LCG2 staging Issue “File not found” (https://ggus.eu/index.php?mode=ticket_info&ticket_id=162827 ); On hold:
  • https://its.cern.ch/jira/browse/EOS-5771
  • Actions from the antares team:
    • A thread was opened in the EOS forum - https://eos-community.web.cern.ch/t/new-citrine-release-4-8-104/892/2 requesting a new EOS citrine version, 4.8.104, that includes the gitlab commit (https://gitlab.cern.ch/dss/eos/-/commit/7c04411e6d6036ae6854dfdf9b0ab4eb319dcdd7) which fixes the issue.
    • Waiting for the CTA project leader and the EOSCTA service manager to be back in office -- ~ two weeks. 
    • Once the new EOS citrine version, 4.8.104, is out it can be pushed to Antares (following some minimum functional testing).
    • No update since last week!

 

• ATLAS jobs failing with: "failed to close file descriptor: bad file descriptor": https://stfc.atlassian.net/browse/GS-131 
  • Creation of a GGUS ticket?
  • Recent (21 August) error on lcg2533:
    • https://bigpanda.cern.ch/job?pandaid=5938012982
    • description: pilot, 1305: Failed to execute payload:FATAL: aborting failed to close file descriptor: bad file descriptor Info:
    • The next checks will be on 24/25 August. 
       
       
• DAOD-on-demand demo test:
  • https://docs.google.com/document/d/118ryBh628w7hSsHPUOjrKvVrnJsm4H_L0DjasvtRlR8/edit
  • Xin Zhao (BNL):
    • "If RAL is ready, we can start the tests at RAL, first step would be to write the sample DAOD 
      datasets to RAL tape. I will create the rucio rules for tape write (and for tape read as well later).
      What we will need from sites will be to clean up tape disk buffer before tape read test, help 
      monitor internal tape situations (e.g. how busy is the staging queue with other production 
      requests etc) and help investigate any site related issues, if any. "

 

• atlas:test file dumps: https://helpdesk.gridpp.rl.ac.uk/Ticket/Display.html?id=485639  [created on 12 July 2023]. 
  • Changes made on 10 August 2023. 
  • davs://rdr.echo.stfc.ac.uk:1094/atlas:test/dumps --> davs://webdav.echo.stfc.ac.uk:1094/atlas:test/dumps/dump_yyyymmdd
  • The latest dump is from 16 August: davs://webdav.echo.stfc.ac.uk:1094/atlas:test/dumps/dump_20230816 (checked last week; this week's is not there yet!)
     

13:20 → 13:21 VO Liaison LHCb

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

slides.pdf

Tickets:

• Failed uploads WN-> ECHO
  • Seems to be caused by network issue
  • Seems to be resolved
• DNS issue
  • No update
• Slow stats
  • No update

Operational issue:

• Farm was drained due to cvmfs issues

 

13:25 → 13:28 VO Liaison LSST

Speaker: Timothy John Noble (Science and Technology Facilities Council STFC (GB))

13:30 → 13:31 VO Liaison Others

13:31 → 13:32 AOB

13:32 → 13:33 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))