RAL Tier1 Experiments Liaison Meeting

→ Europe/London
Access Grid (RAL R89)

Access Grid

RAL R89

Description
    • 13:38 → 13:39
      Major Incidents Changes 1m
    • 13:39 → 13:40
      Summary of Operational Status and Issues 1m
      Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))
    • 13:40 → 13:41
      GGUS /RT Tickets 1m

      https://tinyurl.com/T1-GGUS-Open
      https://tinyurl.com/T1-GGUS-Closed

    • 13:41 → 13:42
      Site Availability 1m

      https://lcgwww.gridpp.rl.ac.uk/utils/availchart/

      https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL

      http://hammercloud.cern.ch/hc/app/atlas/siteoverview/?site=RAL-LCG2&startTime=2020-01-29&endTime=2020-02-06&templateType=isGolden

    • 13:42 → 13:43
      Experiment Operational Issues 1m
    • 13:44 → 13:45
      VO-Liaison ATLAS 1m
      Speakers: James William Walder (Science and Technology Facilities Council STFC (GB)), Dr Tim Adye (Science and Technology Facilities Council STFC (GB))

       - Switching Sheffield to use direct-io Xrootd in certain cases. Had experienced very low CPU efficiency (e.g whole file transfers for Event Index jobs). Should monitor the gateways for any (lack of) vector reads type performance hit.

       

      - Test gateway updated to xrd version 5.0.3 with latest (including Mkdir fix) XrdCeph.

      FTS (dev server issues) - not tested
      Basic upload / download appear ok
      curl_tpc_pull failing (working in v4).

      Smoke test:
      18     UKI-RAL-XROOTD   xrootd-D/ECHO    Of 23 tests: 2 skipped (8%), 21 attempted (91%): 19 successful (90% of tests run, 82% of all tests), 2 failed (9% of tests run, 8% of all tests) Work-arounds: (none)     [in 00:26]
          Initiating an unauthenticated HTTP PULL, authn with X.509 to target: FAILED:  Failed writing received data to disk/application
          Deleting target with X.509: SKIPPED: upload failed
          Initiating an unauthenticated HTTP PULL, authz with macaroon to target: FAILED:  Failed writing received data to disk/application
          Deleting target with macaroon: SKIPPED: upload failed

       

       

    • 13:46 → 13:47
      VO Liaison CMS 1m
      Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))

      CMS went into drain on Monday due to an UNKNOWN status on FTS tests. Because there weren't many transfers to tape, and load tests aren't working yet for tape, there weren't sufficient transfers to give a status over the weekend. As soon as the drain was noticed, it was manually overridden. 

      I'm attempting to tidy up transfer errors coming from a few sites, for various reasons. Some of them are not problems at the RAL end.

      Things are progressing on the slow IPv4 batch farm network. No improvement yet, but a couple more tests done. Martin Bly changed some frame size on Tier 1 ports. DI have plans for a change in early Dec, but no indication this will help my problem. 

      Phedex agents were turned off last Wed. 

      Machine lcgvo06 has been decommissioned. Two scripts from here are now running successfully on cms-rucio-services.gridpp.rl.ac.uk (Echo dump processing and tape approvals). I am working with CMS/Rucio team to make sure the Echo dump file is used for consistency checking

    • 13:48 → 13:49
      VO Liaison LHCb 1m
      Speaker: Raja Nandakumar (Science and Technology Facilities Council STFC (GB))

      LHCb :

      1. SL6 VO-box has been handed back to fabric
        • https://helpdesk.gridpp.rl.ac.uk:443/Ticket/Display.html?id=380695 
      2. ECHO download issue
        • Waiting on software development for xrootd-ceph interface
        • Ongoing progress in routing jobs to LHCb test cluster.

      DUNE:

      Some jobs failing due to network constraint - still trying to understand this. Example query which fails

      https://ifb-data.fnal.gov:8104/ifbeam/data/data?b=DUNE_CERN_SEP2018&t0=1539428280.000&t1=1539428460.000&f=csv

      • Not unique to RAL.
      • User jobs?
      • Issue when multiple jobs start at the same time with this download request
    • 13:52 → 13:53
      VO Liaison Others 1m
    • 13:53 → 13:54
      Experiment Planning 1m
    • 13:54 → 13:55
      Dune/protoDune 1m
    • 13:55 → 13:56
      Euclid 1m
    • 13:56 → 13:57
      SKA 1m
    • 13:57 → 13:58
      AOB 1m
    • 13:58 → 13:59
      Any other Business 1m
      Speakers: Brian Davies (Lancaster University (GB)), Darren Moore (Science and Technology Facilities Council STFC (GB))