RAL Tier1 Experiments Liaison Meeting

Wednesday 2 Dec 2020, 13:30 → 14:30 Europe/London

Access Grid (RAL R89)

Please attend via the following Zoom meeting:

https://ukri.zoom.us/j/98562731547?pwd=UU9Wb2xCL05tWmROT1h6SUlWdUJ3dz09

13:38 → 13:39 Major Incidents Changes

13:39 → 13:40 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))

RT1EL-20201202.docx

RT1EL-20201202.pdf

13:40 → 13:41 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

13:41 → 13:42 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

13:42 → 13:43 Experiment Operational Issues

13:44 → 13:45 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Dr Tim Adye (Science and Technology Facilities Council STFC (GB))

MCTapeDeletionRate.png

WN Slots - Modification to slots count for tranche
 - https://helpdesk.gridpp.rl.ac.uk/Ticket/Display.html?id=381995 

MCTape deletions; slow deletions, affecting overall deletion rate:
 - https://helpdesk.gridpp.rl.ac.uk/Ticket/Display.html?id=382696
Deletions at 0.3Hz, taking 3s per request (suspiciously exactly 3.0s)
Much slower than other T1s (~4Hz); reaper has similar configuration for all T1s

Sheffield;
  Struggling to stage / transfer jobs properly. Likely to be site issue, but should also try to exclude RAL as source of issues.

 

TPC-http;
  Final FTS failure modes understood. due to number "/" being used in URI;
with "//" leads to discrepancy between pool:object name and generated macaroon token name

 

13:46 → 13:47 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

doc1.docx

doc1.pdf

RAL Tape is not being used for CMS as we have hit pledge. There is some cleaning up to do on the CMS side following the Rucio transition. I believe CMS still has spare tape at other sites, so it is not a desperate situation. It was mentioned at the PMB that ~1PB might be offered if CMS requested, so I have passed this on.

Because no files are going to tape, and load tests are not yet up and running, our FTS status is in error. Disk transfers look ok though. We should not go into drain because the status has been manually overridden. 

I asked Rob A about running the Echo file dump ~daily for a few days while the procedure for consistency checking is tested. He agreed in principle, I don't know when this will happen yet, it depends on my FNAL colleague. 

I am keeping in touch with the ongoing CTA migration, and investigating adding load tests. I am thinking about future tests for Production CTA for CMS.

No progress on the IPv4 slowness. However we are running Processing jobs this week and the CPU efficiency has been good. Perhaps the data is all Onsite? But I have no way to properly monitor this and a change to monitoring is not going to happen in the next few months.

13:48 → 13:49 VO Liaison LHCb

Speaker: Raja Nandakumar (Science and Technology Facilities Council STFC (GB))

LHCb:

Streaming from ECHO

• Any news about schedule for fix to xrootd-CEPH plugin?
• Fix to arc-ce05 just rolled out to run jobs on LHCb test nodes only
  • Waiting for stable baseline before proceeding.
  • Want to test out effect of low values for block sizes to see if that alleviates the problem

DUNE

Quite a few jobs in the last week

• But very few right now

ETF testing getting more sites into the "green" territory

• Not all sites in ETF home page are in production

13:52 → 13:53 VO Liaison Others

13:53 → 13:54 Experiment Planning

13:54 → 13:55 Dune/protoDune

13:55 → 13:56 Euclid

13:56 → 13:57 SKA

13:57 → 13:58 AOB

13:58 → 13:59 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))