RAL Tier1 Experiments Liaison Meeting

Wednesday 11 Nov 2020, 13:30 → 14:30 Europe/London

Access Grid (RAL R89)

13:38 → 13:39 Major Incidents Changes

13:39 → 13:40 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))

RT1EL-20201111.docx

RT1EL-20201111.pdf

13:40 → 13:41 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

13:41 → 13:42 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

13:42 → 13:43 Experiment Operational Issues

13:44 → 13:45 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Dr Tim Adye (Science and Technology Facilities Council STFC (GB))

20201112_discuss.pdf

(https://helpdesk.gridpp.rl.ac.uk/Ticket/Display.html?id=375031)
ATLAS appears to suffered from high duplicate file PUTs into castor last week. No similar errors in last 2 days.
ATLAS reports a problem as: "No FTS server has updated the transfer status the last 900 seconds. Probably stalled"
Transfer likely completes on subsequent attempt; but may leave a copy, if the original transfer reports as failed.

ATLAS Tape consistency check for RAL currently being discussed; will accelerate this.

 

 

Brainstorming for "Slow Efficiency Issues"

* ‘Fundamental network / firewall changes’
* Xrootd WN configuration changes
* Ceph changes
* VO software changes
* WN job balance / slot allocation changes
* CVMFS / WN 'network/transfers' issues

?

13:46 → 13:47 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

Screen Shot 2020-11-11 at 12.20.09.png

Tape transfer errors this week, with various messages. Apart from 2 message types, these have cleared themselves up. The two remaining types are:

* Checksum errors (all coming from essentially one site, UCSD. They have fixed some disk failure, but may need to do more before the errors stop).

* Destination file exists errors, which we have seen before, but there are more than 100 unique files producing this error at the moment. The plan currently is to delete them and wait for re-transfer. In fact these files are present and correct on Castor, but there seems to have been some missing part to the completion of the transfer - the PutDone. I'm not sure what else can be done to fix this, except deletion. In trying to figure out what went wrong, we see that this error appeared from the 5th Nov:  'PutDone enforced by internal cleaning'. Rob says that ATLAS have seen a lot of this error too (many more than CMS), and NA62 also see it.

 

The attached image is one I just saw today. This is the read time for 'merge' jobs at CMS T1 sites. Merge jobs should only read Onsite data, e.g. from Echo. The big jump in average read time at the start of July 2020 coincides exactly when I believe the problems around networking started. I previously had no way of monitoring Onsite-only read times, but this plot indicates it may not just be an Offsite read problem after all. 

In combination with the tape transfer errors, some SAM test errors for the AAA, took us into drain for a period on Sunday/Monday. I questioned at the Facilities meeting whether tape write problems should contribute to our Production status. 

 

13:48 → 13:49 VO Liaison LHCb

Speaker: Raja Nandakumar (Science and Technology Facilities Council STFC (GB))

My apologies - in the LarSoft workshop this week

 

LHCb :

Streaming data from ECHO : Waiting for update on xrootd development (Ian Johnson) and grid services team (arc-ce05 fix to stream test jobs only to LHCb test WN group)

New LHCb VO-box : Installed. Waiting to be made dual stack and 3 ports to be opened and alias "voboxlhcb.gridpp.rl.ac.uk" to be moved over to the new machine. Old VO-box stopped.

 

DUNE:

Apparently expect lots of jobs "soon" - currently running around 500 jobs.

13:52 → 13:53 VO Liaison Others

13:53 → 13:54 Experiment Planning

13:54 → 13:55 Dune/protoDune

13:55 → 13:56 Euclid

13:56 → 13:57 SKA

13:57 → 13:58 AOB

13:58 → 13:59 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))