US ATLAS Computing Facility

→ US/Eastern
Description

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  996 1094 4232

Meeting password: 125

Invite link:  https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09

 

 

    • 13:00 → 13:10
      WBS 2.3 Facility Management News 10m
      Speakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
    • 13:10 → 13:20
      OSG-LHC 10m
      Speakers: Brian Lin (University of Wisconsin), Matyas Selmeci

      Impending Release (likely next week)

      • HTCondor 8.9.8 (upcoming) and 8.8.10: prevent condor_qedits from adversely affecting the schedd
      • XRootD 5.0.1 (upcoming)
      • Likely some more EL 8 packages:
        • Blahp
        • HTCondor 8.8.10
        • Frontier Squid
        • LCMAPS
        • XCache
        • XRootD 4
    • 13:20 → 13:35
      Topical Reports
      Convener: Robert William Gardner Jr (University of Chicago (US))
      • 13:20
        TBD 15m
    • 13:35 → 13:40
      WBS 2.3.1 Tier1 Center 5m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
      • Running file in general, several dips in CPU usage due to ATLAS wide issues (CRIC)
      • downtime of dCache is scheduled for August 24~25, for upgrade to v6.2
      • for the ongoing reprocessing, some tails of staging, at the level of O(100), under investigation.
      • singularity will be upgraded to 3.6 on the T1 computing farm, in a rolling fashion.
    • 13:40 → 14:00
      WBS 2.3.2 Tier2 Centers

      Updates on US Tier-2 centers

      Convener: Fred Luehring (Indiana University (US))
      • Generally good running for the last two weeks but a couple of times Hammer Cloud took all US sites offline because of Rucio issues & the CRIC migration.
        • AGLT2 struggled with the knock-on effects from the dCache issues.
        • MWT2 still chasing low level of transfer failure. AC problem solved.
        • NET2 some low level of trouble with oldest systems (older switches?). Enterprise switch issue took site offline briefly.
        • SWT2
      • Please benchmark all new compute servers as they are received.
      • Slate tutorial after this meeting.
      • Need to use Mattermost more
      • 13:40
        AGLT2 5m
        Speakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)

        New hardware purchase:

        - UM put 3 dcache servers online, 2 other servers are about ready soon.
        Put cables on all the C6420 computer nodes, ready for provision.
        The PERC S410 cannot boot from RAID0. Considering options for best usage.

        - MSU received all purchased hardware, storage installation in progress.

        Infrastructure

        Power hickup at MSU on 11-Aug-2020.
        Ground fault on lighting panel made it go offline
        But misconfigured delays made upstream panel also go offline.
        Our circuit is on generator backup, after ~20s.
        Storage and critical networking is behind rack UPS, but not worker nodes.
        For the future: we told that delays have been adjusted.
        Some of our older managed PDUs didn't like the hickup, mainly the recovery.
        We were were already planning on refreshing the older PDUs in the near future.

        Services

        One dcache pool node was failing and shutting down some or all pools at once.
        Have been tracing some errors through our network that we now understand is 100% correlated.
        Working on replacing the cable between 2 main switches upstream.
        This should be solved today, assuming and expecting this was the only problem.

        dCache went offline on last Sunday because two of our zookeeper instances
        having filled their var partitions thus losing quorum.
        We added compression to their logrotate.

        Also getting ready to upgrade to dcache 6.2, starting with upgrading to java11.

      • 13:45
        MWT2 5m
        Speakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))

        UC

        • First batch of compute and auxiliary servers arrived. Rest of compute delayed from Dell, arrival TBD
        • Continuing to debug issues on one of the dCache storage nodes

        IU

        • First batch of compute arrived. Rest of compute also delayed, arrival TBD
        • Continuing to debug IPv6 issues between IU and UC. Appears to be related to ACLs in place on the IU switches
      • 13:50
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))


        Short downtime due to a hardware problem in one of the modules in the BU main N7K.  Fixed over last weekend.  

        Many old nodes repaired with replacement fans.  16 new worker nodes are racked, cabled and powered.  TOR switch replaced on the oldest rack.  

        Preparing to recycle retiring worker nodes and 3TB sleds. 

        New NESE DELL storage is online, cabled with 25Gb/s x 2 and in Ceph.  Will appear in NESE_DATADISK shortly.

        Extensive preparations are underway for "Loch NESE" tape tier.  Space for 133 PB, 50PB ATLAS buy-in, 1PB cache.  Protocols: POSIX and S3.  

        Smooth operations with a full site otherwise.  No tickets.  

      • 13:55
        SWT2 5m
        Speakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))

        OU:

        - Nothing to report, all running well

        - Probably short downtime next week, for upgrading O/S to CentOS 7.8

        UTA:

        • Running smoothly

         

    • 14:00 → 14:05
      WBS 2.3.3 HPC Operations 5m
      Speakers: Doug Benjamin (Duke University (US)), lincoln bryant

      Nothing of significancy to report.

          Debugging of raythena and TACC continues.

      NERSC is running well enough.

      ALCF (Theta) will start soon. - need to use allocation

    • 14:05 → 14:20
      WBS 2.3.4 Analysis Facilities
      Convener: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:05
        Analysis Facilities - BNL 5m
        Speaker: William Strecker-Kellogg (Brookhaven National Lab)
      • 14:10
        Analysis Facilities - SLAC 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:15
        Analysis Facilities - Chicago 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
    • 14:20 → 14:40
      WBS 2.3.5 Continuous Operations
      Convener: Ofer Rind
      • SLATE Tutorial today - 11 registrants
      • A couple of mass "blacklisting" events in the past week due to CRIC issues
      • 14:20
        US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5m
        Speakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
      • 14:25
        Service Development & Deployment 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
    • 14:40 → 14:45
      AOB 5m