US ATLAS Computing Facility

→ US/Eastern
Description

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  996 1094 4232

Meeting password: 125

Invite link:  https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09

 

 

    • 13:00 → 13:10
      WBS 2.3 Facility Management News 10m
      Speakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn Mc Kee (University of Michigan (US))
    • 13:10 → 13:20
      OSG-LHC 10m
      Speakers: Brian Hua Lin (University of Wisconsin), Matyas Selmeci

      Release

      Other

    • 13:20 → 13:50
      Topical Reports
      Convener: Robert William Gardner Jr (University of Chicago (US))
    • 13:50 → 13:55
      WBS 2.3.1 Tier1 Center 5m
      Speakers: Doug Benjamin (Brookhaven National Laboratory (US)), Eric Christian Lancon (Brookhaven National Laboratory (US))
      • Testing of new Tape Library system on going.
      • new storage hardware cabled up and storage system formatting started.  New storage will be added to BNL-OSG2_DATADISK to meet the storage pledge ahead of 1-Apr-22
    • 13:55 → 14:15
      WBS 2.3.2 Tier2 Centers

      Updates on US Tier-2 centers

      Convener: Fred Luehring (Indiana University (US))
      • Good running... Some reduced availability at NET2 and SWT2.
      • Need to check on the current status of IAM server support
        • Missed WFMS meeting today was there any news?
      • My spreadsheet of what we are working on to be ready for data taking:
        • https://docs.google.com/spreadsheets/d/1KniOlqb4dbJ6dKUHBYYt9OfriKjhVpUqXguPvryIMY8/edit?usp=sharing
        • Feel free to update items and items. Please do not delete items. Instead enter a comment that you won't do the item.
      • We need to think about removing gsiFTP but it's still in use at some low level.
      • 13:55
        AGLT2 5m
        Speakers: Philippe Laurens (Michigan State University (US)), Dr Shawn Mc Kee (University of Michigan (US)), Prof. Wenjing Wu (University of Michigan)

            2/17/2022 (ticket 156003)
            All queues of AGL2 site were set to TEST status, because most of the jobs were failing at rucio errors.
            We found out that was caused by a change of the webdav endpoint of for the site.
            We had changed the webdav endpoint from head01.aglt2.org to webdav.aglt2.org
            with a Round Robin DNS entry including our 6 dcache doors.
            We quickly realized the dcache doors did not have the right host certificates
            (missing the subject alternative names in the host certs),
            So we reverted the changes in ATLAS CRIC.
            However, by design, rucio adds but does not automatically delete endpoints without a manual operation.
            The rucio team was short handed, and did not process this manual operation in time,
            so it caused a massive job failure and data transfer failure.

            2/18/2022
            Resolved the problem of the scitoken not working on the gatekeepers
            (the condor-ce variables AUTH_SSL_SERVER_CERTFILE and AUTH_SSL_SERVER_KEYFILE
            were not configured to the right certificate and key files.
            This error is shown in the log file condor-ce/Collector.log, under “will try to use 4096”)
            We then configured all 4 gatekeepers to use sci token for authentication.

            Also upgraded our ATLAS test gatekeeper gate04 from OSG 3.5 to OSG3.6
            (condor 9.0.9-1, condor-ce 5.1.3, only supports scitoken authentication)

            2/20/2022 (ticket 156060)
            We installed new correct hosts certificates for webdav
            and (re-)reconfigured ATLAS CRIC to use webdav.aglt2.org as the service endpoint for webdav protocol.
            Then we got a new ticket about transfer failures due to one host certificate (dcdmsu01).
            After some empirical debugging, it seems the cause is the “full chain” type certificate this host had.
            In fact 5 of the 6 new certificates we received included the full chain of certificates,
            which seems to be fine with most grid services, but not with webdav that failed authentication.
            We truncated the certificate files to just the host cert and that solved the problem.

            2/25/2022
            Nearly 1000 jobs failed because one of our ATLAS gatekeepers gate03 var partition was flooded
            by the condor-ce AuditLog files. The default configuration in condor-ce is to keep 90 days of AuditLog,
            on a busy gatekeeper node, an AuditLog file from one day can accumulate to 2GB.  
            To avoid this from happening again,  reconfigure the value  to 15 days (MAX_NUM_COLLECTOR_AUDIT_LOG = 15).
         
            AMD Benchmark tests:  
            8x R6525 with AMD 7302 3GHz (64HT/system, 256G RAM)  
            Average HS06:     16.42
            Average HEPscore: 17.72

            02-Mar-2022:
            3x R740xd2 with 18 T disks from recent "end of cycle" order arrived at MSU .  
            That will be ~1000 TB usable for dcache.  
            We are still debating if and what we may retire.

      • 14:00
        MWT2 5m
        Speakers: David Jordan (University of Chicago (US)), Jess Haney (Univ. Illinois at Urbana Champaign (US)), Judith Lorraine Stephen (University of Chicago (US))

        UC:

        • Next phase of the data center migration is next week.
        • Taking a downtime on Monday to migrate dcache database
        • 40/80 of the compute from FY21 arrived. Other half soon
        • Waiting to update to dcache 7 until after data center migrations

        IU:

        • Waiting on the new compute order to arrive. Should arrive this week and be installed next.

        UIUC:

        • Emergency maintenance today. Still running jobs from other gatekeepers
        • Permanently shut off mwt2-gk
      • 14:05
        NET2 5m
        Speaker: Prof. Saul Youssef

         

        88 worker nodes arrived; testing.

        New NESE storage and 10 DTNs arrived.  Only waiting for Cisco switches. 

        Upgrading networking between NET2 and NESE. 

        GPFS 3TB pool extracted from file system; retired. 

        NESE tape commissioning with ADC, CMS.

        Converging on plans for new bare metal cluster with new collaborators. 

        Other than bumps extracting 3TB pool, smooth operations.  Site mostly but not always full.  1 GGUS ticket quickly fixed, but took many hours for HC to set us back online.  

      • 14:10
        SWT2 5m
        Speakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))

        UTA_SWT2:

        • Looks like the UTA_SWT2_DATADISK has been officially retired from Rucio as it no longer shows up as an RSE.  Will follow up with DDM-OPS to verify that the disks can be turned off.
        • Getting quotes for moving the racks back to campus and started making space for the equipment.

        SWT2_CPB:

        • Adding a second CE (gk10.atlas-swt2.org) based on OSG 3.5-upcoming.  Configuration is ongoing but starting to work.  Initially we will run Hammer Cloud jobs before moving production jobs to the new CE.
        • Had a problem on the production CE that stopped all work.  Problem is now fixed.  Took the opportunity to rebuild compute nodes and get some configuration work straight for adding the second CE.

        OU:

        - Nothing to report, running well.

         

    • 14:15 → 14:20
      WBS 2.3.3 HPC Operations 5m
      Speakers: Lincoln Bryant (University of Chicago (US)), Rui Wang (Argonne National Laboratory (US))

      TACC 

      • Some recent failures, seems to be related to the payload itself rather than the HPC. Will contact the production folks if it persists.
      • Still at 13% allocation remaining. Have 4 weeks left.

      NERSC

      • Cori running fine
      • Perlmutter jobs running successfully in the test queue. Working on packing jobs similar to TACC, perhaps 16x 8-core jobs to be comparable to grid sites? Queue time limits also a concern. TBD. 
    • 14:20 → 14:35
      WBS 2.3.4 Analysis Facilities
      Convener: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:20
        Analysis Facilities - BNL 5m
        Speaker: Ofer Rind (Brookhaven National Laboratory)
        • A100 hosts have arrived, beginning process to bring them online in next 2-3 weeks
        • Created internal documentation of federated account creation process
      • 14:25
        Analysis Facilities - SLAC 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:30
        Analysis Facilities - Chicago 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

        No major equipment changes for the AF. 

        Continue to work on the portal for ease of navigation, user profile editing, etc.

        Fengping and Lincoln worked on CVMFS install/config - stability improved.

        Farnaz is working on the node-check script.

        Working with Amber on documentation.

    • 14:35 → 14:55
      WBS 2.3.5 Continuous Operations
      Convener: Ofer Rind (Brookhaven National Laboratory)
      • xrootd multiuser plugin validation at BNL is complete (thanks Doug and Qiulan)
      • FedOps:  draft of operations guide under development.
      • BNL VP jobs were not running.  Discovered that repeated core dumps had filled the BNL xcache server boot volume - fixed.  VP queue seems to be running ok since.
      • 14:35
        US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5m
        Speakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
      • 14:40
        Service Development & Deployment 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
        • XCache
          • running fine
        • VP
          • running fine
        • ServiceX
          • logging developments
          • developments for named releases
        • Analytics
          • large development of the new perfSONAR data pipeline underway.
          • last 8 months of  ps_trace data rewritten 
      • 14:45
        Kubernetes R&D at UTA 5m
        Speaker: Armen Vartapetian (University of Texas at Arlington (US))

        I finished setting up Kubernetes on the startup cluster hardware (to start before the main hardware arrives from UTA_SWT2). The K8s cluster works fine and shows healthy. Next is to work on the setup to interface the cluster to run with ATLAS jobs.

    • 14:55 → 15:05
      AOB 10m