US ATLAS Computing Facility

→ US/Eastern
Description

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  996 1094 4232

Meeting password: 125

Invite link:  https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09

 

 

    • 13:00 → 13:10
      WBS 2.3 Facility Management News 10m
      Speakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn Mc Kee (University of Michigan (US))

      The next HEPiX meeting will be April 25-29, 2022.   Please consider putting in abstracts in any of a number of tracks.   https://indico.cern.ch/event/1123214/abstracts/    While site reports are an option, we are involved in many other areas appropriate to report on for HEPiX.

      The #48 LHCONE/LHCOPN meeting concluded today.   Agenda and talks are available at https://indico.cern.ch/event/1110783/

      Please get your quarterly reports in by Friday, April 15. 

    • 13:10 → 13:20
      OSG-LHC 10m
      Speakers: Brian Hua Lin (University of Wisconsin), Matyas Selmeci
    • 13:20 → 13:50
      Topical Reports
      Convener: Robert William Gardner Jr (University of Chicago (US))
      • 13:20
        Postponed till April 13: Storage Consistency Checks at the Tier1 15m

        Talk is postponed till our April 13, 2022 meeting.

        Speaker: Shigeki Misawa (Brookhaven National Laboratory (US))
    • 13:50 → 13:55
      WBS 2.3.1 Tier1 Center 5m
      Speakers: Doug Benjamin (Brookhaven National Laboratory (US)), Eric Christian Lancon (Brookhaven National Laboratory (US))
    • 13:55 → 14:15
      WBS 2.3.2 Tier2 Centers

      Updates on US Tier-2 centers

      Convener: Fred Luehring (Indiana University (US))
      • 13:55
        AGLT2 5m
        Speakers: Philippe Laurens (Michigan State University (US)), Dr Shawn Mc Kee (University of Michigan (US)), Prof. Wenjing Wu (University of Michigan)

        Received and installed the 8 R740xD2 storage nodes for dCache (5 at UM, 3 at MSU), put them in production and rebalanced all the pools. This adds 2.8TB usable space, total usable space token is 10PB. 

         

        03/19/2022 

        The UM Tier3 building had a scheduled power outage, and it was not supposed to affect the status of the Tier3 server room, however water leakage happened in the morning, and the water cooling system stopped working, the server room got overheated, we shut down all work nodes in this room. This problem was resolved in 3 hours(the plumber fixed the leakage, and the clean crew cleaned the water on the floor), and we brought all work nodes back online. Fortunately no damage was made during this incident.

         

        Ongoing ticket with Dell support for R6525.  PERC would sometime not show up after boot.  Traced to PERC data cable at motherboard end. The cable is very close to a metal post and a lid latching hook.  In our case it must have been partially damaged during assembly.  Be careful when you open (and close) the lid that the cable is not in the way of the post.

      • 14:00
        MWT2 5m
        Speakers: David Jordan (University of Chicago (US)), Jess Haney (Univ. Illinois at Urbana Champaign (US)), Judith Lorraine Stephen (University of Chicago (US))

        Intermittent site-wide draining over the past couple of weeks. Still investigating.

        UC

        • The last phase of the server room relocation project was this week. All MWT2 equipment is now in its new data center location.  The last four (relocated) storage servers were brought back online. Currently rebalancing datasets over all pools.
        • Five storage nodes offline today/yesterday due to temperatures (GGUS 156668).   The issues is the inlet temperatures sometimes become elevated due to air circulation.  We are working with the facility manager to address this to find a permanent solution.

        IU

        • Moved IU gatekeeper from UC to IU. Also increased the number of max running jobs on the IU gatekeeper
        • FY22 compute ready to be put online

        UIUC

        • FY22 compute arrived and is in the process of being installed
      • 14:05
        NET2 5m
        Speaker: Prof. Saul Youssef

         

        Two (?) GGUS tickets...

        o Forgot to renew cert.

        o CRL failed to update on xrd-gpfs cluster for unknown reason.

        Both fixed.

        Run 3 preparation:
        1. perfsonar & ipv6 & 88 workers installed but not online yet
        2. OSG 3.6

        NESE Ceph

        New rack arrived except for Cisco top of rack switches.  

        NESE Tape

        Fixed Harvard networking issue.  Dimitrios & Xin helping 50 PB ATLAS space commissioning.  

        NET2 expansion

        Major progress.

      • 14:10
        SWT2 5m
        Speakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))

        UTA:

        • UTA_SWT2 Retirement
          • Cluster is powered off
          • Equipment being moved to campus on 4/1
          • Cleanup of CRIC/OSG Topology on 4/1
        • SWT2_CPB
          • FY22 storage racked (3.2PB usable) burn-in testing ongoing
          • gk10.atlas-swt2.org is new CE (from OSG 3.5-upcoming)
          • Moving old CE to OSG 3.6
          • Test panda queue created for LSM retirement;
            • Works with current Rucio/CRIC configuration using gsiftp
            • Configuration for internal XRootD door broke LSM in standard queue
            • Problem understood; more testing required

        OU:

        - Running well, no issues

        - Had release 22 MCORE jobs running compute nodes out of memory last week, but that was solved by reducing maxrss in CRIC

        - Upgraded SE and backend storage to xrootd 5.4.2 from osg35-upcoming

        - Working on bringing up new CE GK to install OSG 3.6 on.

        - Still waiting for some of the purchased compute nodes from Dell

         

    • 14:15 → 14:20
      WBS 2.3.3 HPC Operations 5m
      Speakers: Lincoln Bryant (University of Chicago (US)), Rui Wang (Argonne National Laboratory (US))
      • TACC: 91% of the 2021 allocation (ends today) has been used
      • NERSC: jobs run smoothly on Cori 
    • 14:20 → 14:35
      WBS 2.3.4 Analysis Facilities
      Convener: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:20
        Analysis Facilities - BNL 5m
        Speaker: Ofer Rind (Brookhaven National Laboratory)
        • BNL A100 installation proceeding: CUDA is set up, burn-in tests running.
          • Benchmarking with MIG shows ~15% performance hit - plan to run this way on some of the GPUs
        • Presentations at last week's HSF AF Forum Kick-off meeting.
        • AF discussion at next Thursday's Snowmass CompF4 Topical Group Workshop
      • 14:25
        Analysis Facilities - SLAC 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:30
        Analysis Facilities - Chicago 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

        New developments:

        • web frontend improvements
        • JuputerLabs get both home directory and data disk mounted
        • working on a brand new monitoring/accounting based on metricbeat/ES/Kibana. Will have everything on one place (condor, coffea-casa, jupyter, space usage).
    • 14:35 → 14:55
      WBS 2.3.5 Continuous Operations
      Convener: Ofer Rind (Brookhaven National Laboratory)
      • One month from OSG 3.5 EOL
      • Yet another major HC exclusion incident yesterday, this time due to problem with one of the Panda servers (aipanda047)
      • Xin presented preliminary results from Tape Challenge today
      • BNL SAM/ETF issues related to XRootd fallback protocol - not yet ready to test gridftp shutoff
      • We are trying to organize a Snowmass pre-meeting discussion about Edge Services this Friday at 1PM ET
      • DevOps:  working on documentation for alert service and admins, need to make sure UTA decommissioning includes squid service (should be handled by topology update)
      • 14:35
        US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5m
        Speakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
      • 14:40
        Service Development & Deployment 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

        Squid:

        XCaches:

        • all working fine

        VP:

        • working fine

        ServiceX:

        • a big reliability upgrade. Will be in the next release 1.0.30 hope to have it released tomorrow, tested and deployed in production by Monday.
        • better logging
        • new features getting tested
      • 14:45
        Kubernetes R&D at UTA 5m
        Speaker: Armen Vartapetian (University of Texas at Arlington (US))

        The startup K8S cluster works fine. I created a queue for it in CRIC, SWT2_CPB_K8S. Waiting for the setup to be completed on the Harvester side. By the time the setup is done, we'll also have more hardware from the UTA_SWT2 move to add to the cluster.

    • 14:55 → 15:05
      AOB 10m