RAL Tier1 Experiments Liaison Meeting

Wednesday 6 Dec 2023, 13:30 → 15:00 Europe/London

Access Grid (RAL R89)

14:00 → 14:01 Major Incidents Changes

14:01 → 14:02 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB)), Kieran Howlett (STFC RAL)

Weekly Report 06 December 2023.docx

Weekly Report 06 December 2023.pdf

14:02 → 14:03 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

14:04 → 14:05 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

14:05 → 14:06 Experiment Operational Issues

14:15 → 14:16 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

Katy at CERN for CMS week.

CMS webdav/xrootd tests are green since Thursday morning when the load balancing parameters were re-tuned. 

DC24 T0 export pre-test was on Thursday. Things went well, although the overall rate to RAL was not higher than previously seen (60-70Gbps). CMS achieved targets for RAL (but not sure about ATLAS). There is a DOMA meeting later today and we will report.

Jobs are doing ok...this morning jobs were pulling in a large amount of data from remote, with a spike of 90+ Gbps (IPv4) seen for the OPN on LibreNMS.

Still seeing the intermittent CVMFS SAM test failures with a couple per day on each CE.

Tokens: Did some more work with Jyothish yesterday but we are still in the same situation where we can't have SAM tests and user tests working simultaneously. The config that passes tests also allows Katy with a user token to access ATLAS etc. data.

Consistency check still failing - the fix tried broke other things. 

Brunel (Raul) is trying to get certificates from RAL but having problems with SAN. 

14:16 → 14:17 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

• GGUS-164289:
  • RAL-LCG2: deletion errors; solved.
  • The amount of timeouts in deletion is relatively stable in the last few weeks (about 4k per day): link-1.
  • RAL was asked to provide an update w.r.t. the remark above.
  • Link-2 --> Deletion Errors --> Deletion Error Overview --> Count (on the right). 
  • Mismatch between expectations and how RAL disk SE actually works.
  • O(3.7k) of these errors per day.
  • RAL is also the fourth most performant site on the grid in terms of deletion successes - 1.15 million in the last week (https://monit-grafana.cern.ch/goto/Uda8imvSg?orgId=17), with ~30k failures.
  • Seems RAL does not have a problem here - it just has an SE that behaves a little differently to others.
  • Closure date: 06 December.

 

• GGUS-164501:
  • UKI RAL: 1.8K errors in production jobs over the last 12 hours; solved.
  • Resolved on 04 December.
  • The tuning of the redirector improved the situation. 

 

• ADC Ops (sic):
  • We wish to notify everyone of the existence of 2 new job queues that have been made live at RAL. These queues, when used, will run jobs in a Rocky 8 or Rocky 9 container environment, respectively. For experiments, please do test jobs against these new queues and report back any issues or questions you may have. This does NOT change the default queue. The default queue for all jobs submitted to RAL is still the “EL7” queue.

 

• DC-24 ramp up challenge (30 November):
  • RAL seemed to have some problem with a really large number of cancelled transfers 34k in 6h and lower rates than expected ( at least as reported by FTS - MONIT sees larger rates).
  • Katy and James can elaborate on this.

 

• Antares IPv4 only:
  • Move to IPv6!
  • Plan to make the EOS nodes for Antares dual stack. 
  • Workernodes shouldn't access tape endpoints but if a site goes IPv6 only, it means the storage will not have IPv4 and that might be a problem.
  • Multi-hop FTS is the way to go?
  • JIRA for IPv6 dual stack enabling: ADCINFR-263.
    

 

• HammerCloud:
  • No exclusions for RAL since 01 December.
  • Hardly any failures after the date. 

 

14:20 → 14:21 VO Liaison LHCb

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

pres_public.pdf

• There was an issue with new xrootd ceph patch roll-out
  • Resulted in all LHCb transfers from ECHO failing for ~1hour
  • Today is another attempt, looks OK so far
• Checksum fix was rolled-out, performance is still not that great (see slides)
• Network issue between RALPP and RAL
  • still present

14:25 → 14:28 VO Liaison LSST

Speaker: Timothy John Noble (Science and Technology Facilities Council STFC (GB))

14:30 → 14:31 VO Liaison Others

14:31 → 14:32 AOB

14:32 → 14:33 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))