RAL Tier1 Experiments Liaison Meeting

Wednesday 19 Aug 2020, 13:30 → 14:30 Europe/London

Access Grid (RAL R89)

13:38 → 13:39 Major Incidents Changes

13:39 → 13:40 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))

RT1EL-20200819.docx

RT1EL-20200819.pdf

13:40 → 13:41 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

13:41 → 13:42 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

13:42 → 13:43 Experiment Operational Issues

13:44 → 13:45 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Dr Tim Adye (Science and Technology Facilities Council STFC (GB))

* lcg2220 and lcg2420 set to ATLAS-only; to compare behaviour seen on ATLAS-only nodes in wn-2016-dell tranche.

* User NS and singularity: GGUS: 146651 
- RAL is last Tier-1 / major site, not to have enabled user namespace;
- Currently affects users with containerised analysis workflow (i.e. would need to replicate data elsewhere if at RAL)

13:46 → 13:47 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

Still waiting to hear from DI about the network intervention. They seem highly uncommunicative, and I have also not seen the ticket submitted by Martin B.

Deletions completed, so it appears - 1.3 PB removed from tape.

Also there were some knock-on effects from Rob's DB intervention on 7th Aug - a bunch of stub files were created, and we saw errors of files trying to re-transfer and failing. These were files managed by Rucio. It took a bit longer to diagnose because Kibana was down (and Phedex looked fine!).

13:48 → 13:49 VO Liaison LHCb

Speaker: Raja Nandakumar (Science and Technology Facilities Council STFC (GB))

LHCb:

1. Small reduction in number of running jobs in the last 24 hours (down to ~7K from ~8.5K)
2. An issue in arc-ce02 & hostcerts
   • Gone away since 15:53 yesterday (18 Aug 2020)
3. Streaming from ECHO
   • https://github.com/xrootd/xrootd/issues/1259
   • Update needed for xrootd-ceph interface
     • Need to prevent each block request from proxy to translate into many (>>>1, up to ~500) requests to CEPH
     • Fixing this likely to help in many ways.
     • C++ development skills needed
   • Possible issue with DELL 2016 WNs - xcache has issues on these WNs
     • Now the dominant source of job failures of this kind.
     • This tranche seems to have issues for ATLAS also
     • Possible options - mulling over (opinions welcome!)
       • Stop LHCb jobs running on that tranche
       • Let the jobs run, but stop the xcache proxy on these WNs alone.

 

DUNE:

Very "peaky" job rates running on the grid (and here)

 

13:52 → 13:53 VO Liaison Others

13:53 → 13:54 Experiment Planning

13:54 → 13:55 Dune/protoDune

13:55 → 13:56 Euclid

13:56 → 13:57 SKA

13:57 → 13:58 AOB

13:58 → 13:59 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))