RAL Tier1 Experiments Liaison Meeting

Wednesday 21 Feb 2024, 13:30 → 15:00 Europe/London

Access Grid (RAL R89)

14:00 → 14:01 Major Incidents Changes

14:01 → 14:02 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore, Kieran Howlett (STFC RAL)

Weekly Report 21 February 2024.docx

Weekly Report 21 February 2024.pdf

14:02 → 14:03 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

14:04 → 14:05 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

14:05 → 14:06 Experiment Operational Issues

14:15 → 14:16 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

DC24 week 2! Rates are higher since yesterday for CMS,  continuing for the rest of the week. Various things done on Echo gateways - network tuning and change to 'ECN' value. 

The main problem at sites including RAL is with the rate of deletions. We had a big spike in the average deletion time this morning. To mitigate, RAL has provided an extra 1PB of disk, and we are trying some thing on the CMS Rucio side related to number of parallel deletions (currently testing at 100).

Jobs are running ok - similarly to other T1s. 

14:16 → 14:17 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

RAL.png

• GGUS-165358:
  • Deletion attempts at RAL-LCG2 take a long time; in progress.
  • Description: the deletions are functional (deleting about 20 TB/hour), but it seems RAL can't keep up with the rate of writing. Looking at details, it seems average deletion duration at RAL is longer that at other sites (some attempts to delete files at RAL take even tens of seconds)!
  • Apart from the status change, no new update from RAL yet!
  • The Echo team is already on it?

 

• Contacted DDM support with regard to move the RAL pledge line to the updated value, i.e., 22.576 PB.
  • Changes made by the DDM folks @ https://s3.echo.stfc.ac.uk/srr/storagesummary.json -- a bit higher.

    • Realised that the values did not account for the 250 TB dedicated to SCRATCHDISK area.

    •  

      Hence, took the values from the pledge in https://wlcg-cric.cern.ch/core/pledge/list/?rc=ral&vo_name=atlas&year=2023&type=disk and adjusted accordingly, considering the +2 PB addition.

  • RAL-LCG2-ECHO_DATADISK.
  • Local changes are already in place (CS-83).
  • Twiki.

 

• ATLAS data carousel (DC24):
  •  The staging traffic subsided last Friday.
  • Yesterday, RAL was removed from the Data Carousel to avoid another flood of staging during the rest of the week.
  • Unfortunately, ATLAS is having problems submitting -- since Monday FTS routine intervention. 
    
  • The situation is being monitored -- when ATLAS will kick in!
    

 

• Reason behind the dip in RAL pledge on 08 February?
  
  • Monit link.
  • Plot also attached.
  • Reason: Panda queue testing.
    

 

• RAL experienced high Panda job failures (~15%) as well as transfer failures ('job canceled by the user') -- as of the morning today.
  

14:20 → 14:21 VO Liaison LHCb

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

dc24_read.png

Transfer_Failures.png

Transfer_Failures.png

Transfer_Throughput.png

Transfer_Throughput.png

DC24:

• Writing part is over
  • 3.96GiB/s was the target rate for RAL
  • Not achieved in general, only for 4 hours we were above it
  • Good efficiency on both links
• Staging started yesterday morning
• Yesterday evening RAL was not achieving its target rate (2.4GiB/s), probably because of the concurrecncy with other VOs
• FTS limit on max running transfers was increased to 300 for Antares -> ECHO link
• Throughput is above the target since then

LHCb:

• No-prefetch sandbox was rolled-out to the whole preprod farm yesterday
• CMS `xrootd-fallback` tests are flapping since then
  
  • These are AAA tests
  • AAA gateways' hostnames are mapped to local GW on every WN
    • Not sure how this is supposed to work, though..

ALICE:

• Investigation of failed tpc test s is ongoing.
  • TCP dump clearly shows that at some point antares server stops closing connections for some reason.
  • EOS developers were unable to hint how to turn off connection reuse.

 

14:25 → 14:28 VO Liaison LSST

Speaker: Timothy John Noble (Science and Technology Facilities Council STFC (GB))

14:30 → 14:31 VO Liaison Others

14:31 → 14:32 AOB

14:32 → 14:33 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore