RAL Tier1 Experiments Liaison Meeting

→ Europe/London
Access Grid (RAL R89)

Access Grid

RAL R89

Zoom Meeting ID
66811541532
Host
Alastair Dewhurst
Useful links
Join via phone
Zoom URL
    • 13:30 → 13:34
      Site Operations 4m
    • 13:34 → 13:35
      Experiment Operational Issues 1m
    • 13:35 → 13:40
      ATLAS Operations Report 5m
      Speakers: Brij Kishor Jashal (Rutherford Appleton Laboratory), Jyoti Prakash Biswal (Rutherford Appleton Laboratory)
    • 13:40 → 13:45
      CMS Operations Report 5m
      Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))
      Thanks to Alex the Stratum-1 issue is fixed (or so it appears in the SAM tests)!
       
      Removing svc06 from the AAA proxy setup. The service is now supported by svc41/42 which should be sufficient. 
       
      Pre-DC27 tests with RAL reported in the DC27 section. 
    • 13:45 → 13:50
      LHCb Operations Report 5m
      Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

      Operational issues:

      • Dirac File Catalogue problem at CERN on the 16th of October
        • DB issues 
        • Caused job failures across the grid, including RAL
      • (not a Tier-1 problem) Were there any reports on network issues between CERN and UK Tier-2 sites, maybe from other VOs?  LHCb observed some problems last night, which looks network related.
    • 13:50 → 13:55
      ALICE Operations Report 5m
      Speaker: Alexander Rogovskiy (Rutherford Appleton Laboratory)

      NTR

    • 13:55 → 14:00
      LSST Operations Report 5m
      Speakers: Thomas Birkett, Timothy Noble (Science and Technology Facilities Council STFC (GB))
      • Odd issues with proxy generation at RAL this is causing some jobs at RAL to fail
        • People at Lancs and SLAC can use SLAC voms server, but we are getting certificate revoked error, yet asking openssl client the cert seems in date
        • Talking to Jyothish I think we got 3B'ed (Technical term, Patent Pending!) - SLAC voms server has a new set of certificates after vendor change and it is using SHA384 (Eliptical curve) and voms-proxy is not handling it well, or it is doing something strange be using the long proxies from the previous cert that have been revoked but site is checking agianst new and crashing, causing the '403' errors
          • Tests from Peter Love swapped to use LANCS voms server to test theory.
            • Tests passed when using LANCS voms for proxy generation, showing the roles etc are correct, and it is something to do with the voms server used
    • 14:00 → 14:01
      Tier-1 Projects 1m
    • 14:05 → 14:10
      XRootD Development 5m
      Speakers: Alexander Rogovskiy (Rutherford Appleton Laboratory), Jyothish Thomas (STFC)

    • 14:15 → 14:20
      Utilizing GPUs 5m
      Speakers: Dr Brij Kishor Jashal (Rutherford Appleton Laboratory), Thomas Birkett
    • 14:25 → 14:45
      Hardware deployment 20m

      Echo SN
      WN
      GPU

      Speaker: Jacob Ward
    • 14:45 → 14:46
      DC 27 1m
      Speaker: Katy Ellis (Science and Technology Facilities Council STFC (GB))
      ATLAS and CMS tested RAL with intention to fill the OPN (200Gbps). We struggled to do this, at first blaming production transfers, e.g. small files taking the I/O, tape recalls etc. But looking more closely at individual file transfer speeds, these seemed to have a distinct split between ‘good’ transfer rate and ‘bad’ transfer rate. During the test, good rate was ~30MB/s. Bad rate was <5MB/s. 
       
      Jyothish tracked down the split to be correlated to the 25G and 100G NIC gateways. The 100s did the good rates, the 25s did the bad rates. Another observation was that the 25s had a huge number of connections - Jyothish tried to correct this by changing the weighting on the decision over which gateway a new connection would be assigned to, but this had little effect. 
       
      Further testing over the following days and using the same system but without the high load of the test confirmed the differences between 25s and 100s. Files were transferred one-by-one, and although the absolute rate was higher (for both gateway types) the difference between them remained very noticeable, e.g. a factor of 5 or more. We do not believe this should be the case. 
       
      Addition of jumbo frames to one of the 25s doubled the rate there (although of course the same addition on the 100s had no effect - as perhaps expected for a short connections like CERN to RAL). So this is not the solution. 
       
      Another observation is that my (slightly dodgy) ‘per-file rate’ monitoring shows a big drop in average file rate on 28th September. However Jyothish has not seen any significant change around that date. 
       
      Only writes affected...reads seem similar across the different gateways.
       
      Jyothish is investigating further...
    • 14:50 → 14:59
      Summary of Operational Status and Issues 9m
      Speakers: Brian Davies (Science and Technology Facilities Council STFC (GB)), Darren Moore, Thomas Birkett
    • 15:00 → 15:05
      Any other Business 5m
      Speakers: Brian Davies (Science and Technology Facilities Council STFC (GB)), Darren Moore
    • 15:15 → 15:16
      AOB 1m