RAL Tier1 Experiments Liaison Meeting

Wednesday 24 Jun 2020, 12:30 → 13:30 Europe/London

Access Grid (RAL R89)

12:38 → 12:39 Major Incidents Changes

12:39 → 12:40 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))

RT1EL-20200624.docx

RT1EL-20200624.pdf

12:40 → 12:41 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

12:41 → 12:42 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

12:42 → 12:43 Experiment Operational Issues

12:44 → 12:45 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Dr Tim Adye (Science and Technology Facilities Council STFC (GB))

Problems ( #292139) recovering from Power outage / singularity update; ;
  James rollbacked changes ;    MCORE queue now was online; now set test again. GU queue remains in test.

ATLAS now switched away from RAL-FTS.

Major CERN DB intervention planned for 27th June (Sat), ATLAS affected (jobs / transfers) can expect to be affected.

12:46 → 12:47 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

Low availability for transfers starting on Friday and over the weekend due to Network issues. Resolved on Sunday. Communication with Martin Bly:

Katy Ellis  11:43 AM

hello Martin! Quick question about the weekend network problem. You said at the 9am meeting that this outage started Saturday into Sunday. However, I saw in monitoring that transfers started to fail intermittently on Friday. Is it that the network failed gradually, or that there were multiple issues? For example I heard mention of the RAL FTS

Martin B(OfH)  12:17 PM

So on Thursday 18th at about 9:30 there was a wobble on one of the circuits to Cern due to 'low light levels' in the optical fibres - this is a Janet issue and is in principle being fixed by changing fibres today.  The OPNR saw the wobble in its logs.This may have caused the OPNR to have it's own wobble triggering the problems we had but this is difficult to say for sure.At 02:21 UTC on Friday the OPNR starting reporting frequent issues with it's internal memory allocation activities to do with BGP and other routing stuff.   At 21:10 UTC same day it reported an out of memory error, the first of O(lots). This continued through Saturday 20th and Sunday 21st.  (Switch timezone is UTC, add 1 hour for BST.)The other switches report the link to the OPNR from the Tier1 core started wobbling up and down at 21:10 UTC on Friday with a stable period from 00:20 UTC Saturday to 08:41 UTC Sunday at which time it wobbled up and down.At about  09:30 on 21st the OPNR stopped responding to queries from the traffic monitoring system LibreNMS and continued sulking until about 15:10 when it woke up again. Traffic was at that point passing normally, but hadn't as far as was known been working prior to that.I poked the OPNR at about 16:15 21st to reload (reboot) it, and it has been up and stable since, with no reports of memory issues.FTS would have been affected due to the data traffic all departing and arriving on site through the OPNR, either from Janet (firewall bypass) or Cern (LHCOPN).

----------------

There was also a power outage on Monday. CMS managed to keep running some cores. More cores came back on Tuesday afternoon/evening. CMS is now running ~5k cores according to CMS monit, but ~10k cores according to Vande - monitoring discrepancy?

 

12:48 → 12:49 VO Liaison LHCb

Speaker: Raja Nandakumar (Science and Technology Facilities Council STFC (GB))

LHCb

1. Recovering from outage
2. ECHO streaming testing
   • Understand reason why moving from disk to memory proxy broke the system
   • Currently test system broken on LHCb end
     • Authentication not passed along to WN
     • ECHO access fails as a result (permission denied)
3. New ARC CE testing
   • Multiple issues - fixing one by one
4. Shutting down DIRAC services on Friday for the CERN dB downtime

DUNE

No special news

• Jobs are running - the power cut was noticed

12:52 → 12:53 VO Liaison Others

12:53 → 12:54 Experiment Planning

12:54 → 12:55 Dune/protoDune

12:55 → 12:56 Euclid

12:56 → 12:57 SKA

12:57 → 12:58 AOB

12:58 → 12:59 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))