US ATLAS Computing Facility

→ US/Eastern
Description

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  996 1094 4232

Meeting password: 125

Invite link:  https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09

 

 

    • 13:00 → 13:10
      WBS 2.3 Facility Management News 10m
      Speakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
    • 13:10 → 13:20
      OSG-LHC 10m
      Speakers: Brian Lin (University of Wisconsin), Matyas Selmeci

      * OSG 3.5.21 (last week)
        * HTCondor-CE 4.4.1
          * Fixed a race condition that could result in removed jobs being put on hold
        * Limited EL8 support:
          * Worker node client (RPMs, tarballs, containers)
          * OASIS/CVMFS

      * OSG 3.5.22
        * Gratia Slurm probe fixes
        * EL8:
          * HTCondor
          * CE packages
          * XCache

    • 13:20 → 13:35
      Topical Reports
      Convener: Robert William Gardner Jr (University of Chicago (US))
      • 13:20
        TBD 15m
    • 13:35 → 13:40
      WBS 2.3.1 Tier1 Center 5m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
      • normal operation in general 
      • reprocessing running fine, BNL staging rate for data17 is 4GB/s on average
      • weathered the tropical storm well, all services were not affected. 
    • 13:40 → 14:00
      WBS 2.3.2 Tier2 Centers

      Updates on US Tier-2 centers

      Convener: Fred Luehring (Indiana University (US))

      Lots of news:

      • Each Tier 2 getting about $100k from NSF for FY21.
      • Got generally full amounts for infrastructure/networking too... (Srini end of period money)
      • Still waiting for Tier 2 review report.
      • Generally good running for the last two weeks.
        • AGLT2 traced transfer errors ipv6
        • MWT2 still chasing low level of transfer failure. AC problem crept up at UC over last ~3 weeks.
        • NET2 some low level of trouble with oldest systems
        • SWT2 deletion issues at CPB
      • Please benchmark all new compute servers as they are received.
      • Slate tutorial needs to be scheduled
      • Need to use Mattermost more
      • Ofer get the info he needed about the mechanism for marking downtimes?
      • CRIC migration planned to be completed in September see latest report (July 13): https://indico.cern.ch/event/936150/contributions/3933422/
      • 13:40
        AGLT2 5m
        Speakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)

        No open ticket

        UM received all 2020 new purchased hardware, 5 R740x2d, 20 C6420, 80% of them are on the rack, and we are in the process of provision these nodes.
        MSU bureaucracy was slower but POs issued, waiting on delivery

        dcache update to 5.2.25:
        made us realize and find IPv6 problem, fixed.
        still sporadic dcache problems with pools setting themselves offline after detecting consistency problems. Forwarded logfiles to dcache ticket

        afs: updated all AFS servers from SL6 to CC7, and afs server to 1.8.6

         
      • 13:45
        MWT2 5m
        Speakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))

        UC

        • Debugging temperature issues in the server room
          • SE set offline over the weekend due to servers overheating and shutting down
          • UIUC and IU workers fully online now, bringing UC up incrementally
        • Equipment POs submitted 

        IU

        • Equipment POs submitted

        UIUC

        • A couple workers down for ongoing hardware issues, otherwise running as usual
      • 13:50
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))

         

        Making final preparations for 6PB expansion of NESE_DATADISK space in NESE/Ceph.  Existing NESE deployment has been running fine for many months now.  

        We're making the final preparations before ordering the first NESE Tape Tier purchase, deployment and launch.  After extensive interviews with BNL, LANL, NERSC, TACC, NCSA, MIT, we've decided on a 9 frame, IBM TS4500 (same model that BNL is getting), with full capacity of 140PB.  The front end will be a 1PB FC attached storage running IBM Spectrum Scale/Spectrum Archive ("GPFS").  Protocols will be POSIX & S3 via MinIO.  I/O will initially be 11GB/s initially, ramping up to 17GB/s as we add more drives.  MGHPCC has reserved an entire pod for this with space for a total of 4 libraries of this size. 

        Preparing for a major trip to MGHPCC to take care of quite a few backed up issues:  Installing new workers, replacing fans, fixing broken disks, re-cabling and replacing a likely failing 15 year old TOR switch. 

        Operations have been smooth with the site fairly consistently full. 

         

      • 13:55
        SWT2 5m
        Speakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))

        OU:

        - Running well, no issue

        - There were problems last week where arturos AP_REPR jobs were consuming MUCH more RAM than they had requested, which took down quite a few of our nodes because they ran out of RAM. That was apparently caused by scout jobs reporting incorrect (much too low) RAM usage on these jobs, which was eventually corrected. Paul and others are investigating how to prevent this from happening in the future.

        - Updates from last time:

         - OSG downtime only causes Switcher to turn sites of if OUTAGE is chosen, not SEVERE or other options.

         - SAM tests have been modified to only request 10 minutes of walltime now.

        UTA:

        => Hardware from the most recent round of purchases is being delivered. 

        => 8/4 - 8/5: Filesystem mount on one of the gridftp hosts hung up, causing file deletion errors. Fixed - closed the ggus ticket.

        => 7/26: Problem with the chilled water supplying the CRAC's at SWT2_CPB. Facilities responded quickly - A/C's returned to normal temps after ~one hour.

        => Otherwise smooth operations over the past two weeks.

    • 14:00 → 14:05
      WBS 2.3.3 HPC Operations 5m
      Speakers: Doug Benjamin (Duke University (US)), lincoln bryant

      NERSC is running albeit slowly - 57 days till empty. 

      Scale testing at TACC is proceeding slowly.

      Testing new image (release 21.0.15 gcc4.9) 

         Note: we are running release 21.0.15 gcc 6.2 image.  

    • 14:05 → 14:20
      WBS 2.3.4 Analysis Facilities
      Convener: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:05
        Analysis Facilities - BNL 5m
        Speaker: William Strecker-Kellogg (Brookhaven National Lab)
      • 14:10
        Analysis Facilities - SLAC 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:15
        Analysis Facilities - Chicago 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

        all working fine. Working on updating python, cuda, drivers.

    • 14:20 → 14:40
      WBS 2.3.5 Continuous Operations
      Convener: Ofer Rind

      No big news.  Some discussion at DOMA-TPC meeting this morning re: updates to XRootD 5 for HTTP TPC testing at OU and SLAC.

      • 14:20
        US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5m
        Speakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
      • 14:25
        Service Development & Deployment 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

        SLATE

        deployment at OU: Patrick will continue with it after coming back from vacation (10th Aug).

        NET2 Saul should be working on it now.

        VP

        Still in broker-off as the release on 97a is still not there. (Ed still on vacation)

        Work on "fully diskless" PQ still blocked waiting for explanations from Tadashi/Fernando (for few months already). 

        XCache

        using OSG base at 5.0.0-1

        Analytics 

        running fine

    • 14:40 → 14:45
      AOB 5m