Focus on:  Hide Contributions
Login

US ATLAS Tier 2 Technical

→ US/Eastern
Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), Shawn Mc Kee (University of Michigan (US))
Description

Meeting to discuss technical issues at the US ATLAS Tier 2 site. The primary audience is the US Tier 2 site administrators but anyone interested is welcome to attend.

Fred Luehring
US ATLAS Tier 2 Technical
Zoom Meeting ID
67453565657
Host
Fred Luehring
Useful links
Join via phone
Zoom URL
    • 11:00 → 11:10
      Introduction 10m
      Speakers: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

      Quick links:

      General news

      • Running over the past two weeks has been good.
        • NET2 got about 3k job slots back in production on 26-June
        • NET2 had an average wall time job success efficiency of about 87.5%
        • OU is not showing any production since storage issues 2-July on the site oriented dashboard. 
        • TW-FTT had a small reduction in job slots from 1-July to 3-Jul.
      • The area 2.3 scrubbing is next Tuesday 14-July
        • I am close to ready and will circulate the slides hopefully today for final comments.
      •  I need each site's  quarterly reporting by the end of Friday July 17.

      Upcoming meetings:

      Open tickets:

       

    • 11:10 → 11:20
      TW-FTT 10m
      Speakers: Eric Yen, Felix.hung-te Lee (Academia Sinica (TW)), Yi-Ru Chen (Academia Sinica (TW))
      • There was a 3-hour network provider circuit interruption on July 1.
      • We adjusted the CPU cores allocation for the multi-core job from 12 to 24 on July 2.
    • 11:20 → 11:30
      AGLT2 10m
      Speakers: Daniel Hayden (Michigan State University (US)), Philippe Laurens (Michigan State University (US)), Shawn Mc Kee (University of Michigan (US)), Dr Wendy Wu (University of Michigan)

      Security updates
        Applied all known mitigations
        MSU WNs updated last week to kernel-5.14.0-687.20.1.el9_8 
        Just noticed  RHSA-2026:36645 with kernel-5.14.0-687.23.1.el9_8
          which includes fix for "Bad epol" CVE-2026-46242
          not yet on our RH satellite provisioning systems
        Will update dCache and other service nodes this week or next
        Evaluating options for UM WNs
          as Lustre is not currently supporting RHEL 9.8, while support not expected soon

    • 11:30 → 11:40
      MWT2 10m
      Speakers: Aidan Rosberg (Indiana University (US)), David Jordan (University of Chicago (US)), Farnaz Golnaraghi (University of Chicago (US)), Fengping Hu (University of Chicago (US)), Fred Luehring (Indiana University (US)), Judith Lorraine Stephen (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
      • Testing CVMFS 2.14.0~pre5-2.osg25up.el9 on a subset of IU nodes. Noticed some reboots. Aidan is working on logging and debugging

      • IU and UC still working on network and hardware refresh plans

      • Waiting for kernel-5.14.0-687.20.3.el9_8 to be released for the epoll cve

      • UIUC quarterly maintenance is scheduled for July 15th

    • 11:40 → 11:50
      NET2 10m
      Speakers: Eduardo Bach (University of Massachusetts (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), William Axel Leight (University of Massachusetts Amherst)

      Since the downtime, issues with jobs failing due to diskIO errors and insufficient storage.  Some tuning on the dense servers to handle diskIO was needed before they could all be added to the cluster, so we did not get back to the nominal number of slots immediately.  We are working on replacing or upgrading disks in some servers to help with the diskIO issues, and will also investigate tuning the Maxdiskio parameter as Rod suggested in his email this morning.  The previous cause of insufficient storage errors, the pilot getting the available space on the entire server and not the pod, was fixed some time ago, so it's still unclear why these errors have returned.

      Had to restart storage servers Monday to apply security mitigations.  This led to two internal dCache cells becoming problematic, causing successful tape recalls to be reported as failed.  The result was several waves of failed transfers as tape staging jobs were repeatedly retried, until the problem was identified and fixed.

    • 11:50 → 12:00
      SWT2 10m
      Speakers: Andrey Zarochentsev (University of Texas at Arlington (US)), Horst Severini (University of Oklahoma (US)), Kaushik De (University of Texas at Arlington (US)), Mark Sosebee (University of Texas at Arlington (US)), Zachary Thomas Booth (University of Texas at Arlington (US))

      SWT2_CPB: 

      • GGUS-Ticket-ID: #1003078 - SWT2_CPB transfer failures as source with File Not Found error

        • Received replacement PERC to install in storage where we took the PERC from for faster repair. 

        • Replaced the problematic drive in this storage to rebuild.

        • We are currently running diagnostic tests on both storage servers involved, rebuilt, and are performing other tests after repairs to check for any additional issues. 

        • Storage server has been stable since PERC replacement. 

      • GGUS-Ticket-ID: #1003053 - IGTF CRLs & fetch-crl SHA1 signature validation (SWT2_CPB)

        • We requested this XRootD Proxy server be removed from our DNS round robin two weeks ago, but they were able to remove it yesterday.  

        • This reduces the number of XRootD Proxy servers we have active down to three, and allows us to check this server more thoroughly and rebuild. 

        • The last transfer error we experienced due to this was yesterday, since this DTN was just removed then. 

      • We experienced some brief transfer errors throughout the past two weeks as we worked toward upgrading the network of the XRootD Proxy server racks. 

      • We rebuilt the last ME4084 storage server to EL9. It was temporarily out of service with no real data (not in production). 

      • We changed the “Monitored” parameter for our test cluster to “True” in CRIC. 

      • We experienced some issues with jobs due to the high number of single core jobs on 7/4. 

        • We believe this is due to our Slurm max job limit, and slightly increased this to from 12000 to 13000 to see if the issue resolves. We may gradually increase this over time to fix if needed. 

      • CVE-2026-55200 does not apply to us, so no action taken. We have a different libssh2 version. 

      OU:

      • Back up from scheduled maintenance, but at reduced capacity, until OSCER has had time to re-image all nodes.
      • New SE, OU_OSCER_ATLAS_SE_1 should be up now, so assuming that works, we will need to migrate from the old and out of date SE to this new one as fast as possible, because of new stricter OU IT security rules.