RAL Tier1 Experiments Liaison Meeting

Wednesday 15 Nov 2023, 13:30 → 15:00 Europe/London

Access Grid (RAL R89)

14:00 → 14:01 Major Incidents Changes

14:01 → 14:02 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB)), Kieran Howlett (STFC RAL)

Weekly Report 15 November 2023.docx

Weekly Report 15 November 2023.pdf

14:02 → 14:03 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

14:04 → 14:05 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

14:05 → 14:06 Experiment Operational Issues

14:15 → 14:16 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

Katy has been away for several weeks on leave and at various workshops.

In general, job performance has been good over recent weeks.

A small number of file transfer failures discovered today, from Antares to Echo on the 8th Nov - coincides with the SN problem (hardware was retired). 

A few weeks ago there was a downtime for electrical work over a weekend, and a several day long downtime. In general CMS respected the downtime as expected, but there was one job type, which is a new type running jobs that would normally run at T0 at the T1s, which continued to submit jobs. I have pointed this out to T0 team and they have reassured me they acknowledge they need to respect site downtimes and won't do it again.

CMS ARM testing at Glasgow is almost complete, but it appears that CPU efficiency was very low. Katy will investigate and report for the benefit of Glasgow and other UK interested parties. 

CMS Tier 1 transfer tests between CERN and T1 disk sites: Each T1 has now been tested multiple times. Several sites reached their expected throughput according to their network (OPN) provision. Others needed some work. At RAL we (me, XRootd team people, Echo team people) had a look at the outcome and postulated that there is a 'limit' or preference on single file transfer rate of ~5MB/s. This seems low. Katy to confirm and report back to the group. 

Just spotted interesting observation - majority of jobs running in the last month are labelled 'Offsite reads'. This means something specific - that the "primary" input dataset is offsite, and the intention is to read it offsite. This does NOT count premix library reads. These jobs are Processing type 8-core jobs using ~10GB. The amount of data being read (probably from CERN) per job is quite small. Another interesting observation is that the number of completed jobs of this type is very small - jobs have very long run time, but performance is mostly good (could use some investigation).

Change control tomorrow for token roll-out?

Some good indication that Shoveler is working ok on the AAA machines (reading data) at RAL. I've seen the Shoveler monitoring and it has a similar scale and shape to our AAA monitoring in Vande. I will look at it again over some longer time periods. The previous issue before with the Shoveler service losing connection with the collector (but appearing like it continued to run) seems to be solved, and I have not needed to restart the service in a long time.

14:16 → 14:17 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)

RAL_jobs_2days.jpg

• https://ggus.eu/index.php?mode=ticket_info&ticket_id=164115:
  • Partial network issue between RAL-LCG2 and FZK; waiting for reply. 
  • The new advertisement of RAL is now added to the GridKa Border Router configuration The issue should be solved. --> Verification awaited from RAL.

 

• RAL has been lacking jobs since the past two days (plot attached).
  • The situation has improved in the past 12 hours, but still below the pledge. 
  • Lack of jobs @LHC in general?

 

• There were several HammerCloud exclusions for RAL recently.
  • Echo/XRootD-related.
  • For short time periods. 
  • None since yesterday. 

 

• In the past 24 hours, the number of failed jobs is more than 50% of the number of finished jobs.

 

14:20 → 14:21 VO Liaison LHCb

Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

Issues:

• Approximately 250 files were lost 08.11, due to ECHO incident.
  • Seems like this loss happened due to xrootd retries
  • I requested removing xrootd from the list of RAL write protocols
    • For Anares -> ECHO transfers xrootd will still be used
• A lot of failed downloads from ECHO to local WNs
  • Periodically local WN proxies got "stuck" for some reason
• There were a few spikes of failed uploads yesterday afternoon
  • Probably due to ECHO issues
  • All generations were affected
• Antares head restarts happen periodically, resulting in a few FTS failures
  • It is interesting that most of the affected transfers were in "ARCHIVING" state, so files were already copied to their destination
  • Maybe multiple poll attempts may help
• Network issue between RALPP and RAL seems to be still present
• Seems like new EL8 and EL9 queues can not be used by LHCb so far
  • Pilots execute successfully, but do not pick up any jobs
  • Not clear what is happening so far

14:25 → 14:28 VO Liaison LSST

Speaker: Timothy John Noble (Science and Technology Facilities Council STFC (GB))

14:30 → 14:31 VO Liaison Others

14:31 → 14:32 AOB

14:32 → 14:33 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))