RAL Tier1 Experiments Liaison Meeting

Wednesday 5 Aug 2020, 13:30 → 14:30 Europe/London

Access Grid (RAL R89)

Please attend via the following Zoom meeting:

https://ukri.zoom.us/j/98562731547?pwd=UU9Wb2xCL05tWmROT1h6SUlWdUJ3dz09

 

13:38 → 13:39 Major Incidents Changes

13:39 → 13:40 Summary of Operational Status and Issues

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))

RT1EL-20200805.docx

RT1EL-20200805.pdf

13:40 → 13:41 GGUS /RT Tickets

https://tinyurl.com/T1-GGUS-Open
https://tinyurl.com/T1-GGUS-Closed

13:41 → 13:42 Site Availability

https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

13:42 → 13:43 Experiment Operational Issues

13:44 → 13:45 VO-Liaison ATLAS

Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Dr Tim Adye (Science and Technology Facilities Council STFC (GB))

 

* Reprocessing 2017 data
  2017 staging of reprocessing campaign started Monday evening. 

* Deletions from tape
  Preference to start next week (once expert back from vacations);
Can this fit with migration timescale?

 

* Direct-io/streaming errors:
    "TNetXNGFile::Close        ERROR   [ERROR] Invalid session"
Pile-up ntuple making tasks; using direct-IO.
Reads a small fraction of data from each event.
Seen in multi-file (O(10GB/file) jobs
 - Appears to be quite similar to issues that LHCb have previously observed?

Fatal, as Log-parser script finds the unknown "ERROR" message; but job does appear to run OK.
 

* IPv4/6 speed differences:
Seem to confirm (corroborating  observations already from CMS), that copies into RAL over IPV4 can be O(10x) slower than equivalent IPV6 transfers. (cf O(5MB/s) ipv6 vs O(400kB/s) ipv4 using gridFTP and Xrootd.)
 

 

13:46 → 13:47 VO Liaison CMS

Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

Job efficiency still very low, dominated by Processing-type events. Many jobs do have the Offsite tag. I believe this only applies to Primary inputs though - secondary inputs may be offsite and not marked in this way. My suspicion is that 'most' jobs have some kind of offsite data streaming. 

I did some tests this week - copy a file using gfal-copy from offsite to lcgui06, specifying IPv4 or IPv6 in each case. Example - copying file from Nebraska - took 46 seconds with IPv6 and 1725 seconds with IPv4. The same file copied with destination on mercury (RAL PP) took 47 seconds with IPv4. Saw similar thing with file at FNAL.

James W tried a similar test for ATLAS, with a file from Tokyo, and saw something quite similar. Raja tried for LHCb with a file from Germany and did not see the same thing.

Rate from Echo looks good with either IPv4 or 6.

There is a ticket with DI. We did a trace route to Nebraska, and got Nebraska to do the reverse route back. However, Martin B said:

The reverse traceroute gives no useful information after it hits the boarder inbound due to the site insistence on blocking ICMP packets except in specific cases.  That's fatal to IPv6 doing it's thing with path MTU discovery, but should not affect IPv4 traffic - just makes it a pain to diagnose stuff. 

Phedex transfers also look 'patchy' since Friday - a mix of failures and successes from many sites, particularly T2s. Not sure if this is related - would need to see what is going over IPv4/6.

13:48 → 13:49 VO Liaison LHCb

Speaker: Raja Nandakumar (Science and Technology Facilities Council STFC (GB))

LHCb

• ARC CEs running fine on v6 as far as LHCb is concerned
• LHCb streaming issue
  • No particular change in job failure rate with new configuration of block sizes at 64MB
  • Possible dependence of "Watchdog identified this job as stalled" on number of concurrent running jobs
    • More correctly the waiting jobs. Do not have a plot of running jobs
    • Details on wiki page
    • Note - these changes were only on the 6 LHCb test WNs and not across the batch farm
  • arc-ce05 currently draining.
    • Some traceback for the xrootd developers after it is back (Tom)

 

DUNE

Nothing special to report. Very few running jobs.

13:52 → 13:53 VO Liaison Others

13:53 → 13:54 Experiment Planning

13:54 → 13:55 Dune/protoDune

13:55 → 13:56 Euclid

13:56 → 13:57 SKA

13:57 → 13:58 AOB

13:58 → 13:59 Any other Business

Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))