RAL Tier1 Experiments Liaison Meeting

Wednesday 12 Aug 2020, 13:30 → 14:30 Europe/London

Access Grid (RAL R89)

Please attend via the following Zoom meeting:

https://ukri.zoom.us/j/98562731547?pwd=UU9Wb2xCL05tWmROT1h6SUlWdUJ3dz09

 

13:38 → 13:39 Major Incidents Changes

13:39 → 13:40 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))

RT1EL-20200812.docx

RT1EL-20200812.pdf

13:40 → 13:41 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

13:41 → 13:42 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

13:42 → 13:43 Experiment Operational Issues

13:44 → 13:45 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Dr Tim Adye (Science and Technology Facilities Council STFC (GB))

* Data reprocessing campaign
2017 staging gone well
Expect 2015+16 to start staging early next week

* ATLAS Started Tape deletions; 6k deletions /hr; 2PB total volume. Two weeks expected
- Rate can be adjusted if needed.

* Rucio update (to 1.23.0) affected staging, (early Tuesday). Version rolled back by afternoon.

* User NS and singularity: GGUS: 146651 
- RAL is last Tier-1 / major site, not to have enabled user namespace;
- Currently affects users with containerised analysis workflow (i.e. would need to replicate data elsewhere if at RAL)
  

 

 

13:46 → 13:47 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

Continuing to push for the very slow transfers to be investigated by Network team/Martin B. Job efficiency is very variable. I have asked CMS to better define 'Offsite' jobs, but no idea when this will happen.

CMS FTS tests are failing. I have written to the person who set up the monitoring for help understanding. Phedex monitoring shows some failures, but not as bad as the test shows.

We are doing a deletion totalling 1.3PB from RAL Tape. This started yesterday. I am approving it in chunks, as requested. 

13:48 → 13:49 VO Liaison LHCb

Speaker: Raja Nandakumar (Science and Technology Facilities Council STFC (GB))

LHCb:

Streaming issue from ECHO :

• Lots of activity in github issue : https://github.com/xrootd/xrootd/issues/1259
  • TL;DR (from my limited understanding) :
    • Hitting 350s timeout for retrieving 39GB of data from CEPH for an example 9MB read request from application
    • Requests get passed through directly to CEPH, without going through cache, when RAM "runs out"
    • Changes need to be safe for the alternate workflow of copying data to the WNs
• Sending monitoring info to UCSD also now
• Recently most failures are dominantly in 2016 Dell WNs
  • Various updates have happened since the previous check
  • SSD may be helping (not completely clear, as 2014 WNs seem okay)

DUNE:

Low level of running jobs over the last week, though peak of ~2k jobs in the last day.

 

13:52 → 13:53 VO Liaison Others

13:53 → 13:54 Experiment Planning

13:54 → 13:55 Dune/protoDune

13:55 → 13:56 Euclid

13:56 → 13:57 SKA

13:57 → 13:58 AOB

13:58 → 13:59 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))