US ATLAS Computing Facility

→ US/Eastern
Description

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  996 1094 4232

Meeting password: 125

Invite link:  https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09

 

 

    • 13:00 → 13:10
      WBS 2.3 Facility Management News 10m
      Speakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
    • 13:10 → 13:20
      OSG-LHC 10m
      Speakers: Brian Lin (University of Wisconsin), Matyas Selmeci

      OSG 3.5.31

      • osg-ca-certs 1.94: adding the cross-signing Let's Encrypt root CA
      • osg-release 3.5-5: preparation for separate OSG 3.5 upcoming repositories
      • CVMFS 2.8.0
      • XRootD 4.12.6
      • SciTokens python library 1.3.1

      Other

    • 13:20 → 13:35
      Topical Reports
      Convener: Robert William Gardner Jr (University of Chicago (US))
    • 13:35 → 13:40
      WBS 2.3.1 Tier1 Center 5m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
      • Smooth operation in general
      • dCache restarted yesterday, as a workaround for the transfer issues with Swiss sites caused by a bug in the CaNL library, triggered by a recent IGTF CA release.
      • Thorough scanning and analysis on space usage in dCache was conducted, now the cause for the space reporting discrepancy is understood, biggest factor is duplicated file copies on the primary pools. The issue is fixed
    • 13:40 → 14:00
      WBS 2.3.2 Tier2 Centers

      Updates on US Tier-2 centers

      Convener: Fred Luehring (Indiana University (US))
      • Reasonably stable running except NET2 where some sort of GPFS problem caused an outage:


      • Network load on the CERN side - from today's ADC Daily minutes:
        • Rucio 
          • (Martin) All the missing rules from the subscriptions problem created
        • Transfers 
          • Transfers peak above 60GB/s due to Data Consolidation from above point
      • Noticed at AGLT2 & MWT2.
      • Requested for the Monit plots generated that TACC be moved out of MWT2 and into the US Tier 3 sites where the other supercomputers are.
        • Also got the countries USA and United States of America combined into being just the USA
      • Still need to wrangle with some other naming conventions.
      • Still trying to solve stuck transfers issue.
      • Agreed to help out Frontier-Squid Dev team
      • Sysadmins should attend the ADC Weekly round table.
      • Need to keep up with IPV6, HTTP-TPC, and OSG 3.5.
      • Spend your network money by the end of February - that's a milestone.

         
      • 13:40
        AGLT2 5m
        Speakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)

        1 ggus ticket: 5 missing files. File were lost from hardware failure last Oct. They were not included in the original lost file we declared. Maybe because they did not have metadata in the chimera database. 

        1 ggus ticket about local muoncal jobs using the proxy server at CERN instead of aglt2 proxies, this is due to a bug in Frontier Server setup, Asoka fixed it. 

        Network issue:

        2 weeks ago we reported a problem with the UM-MSU 2x40G link.

        This was tracked down and solved by replacing one 40G transceiver in the MSU computing room AND rebooting the juniper line card hosting that port. The link was restored to full capacity 21-Jan-2021

        Early this morning we had a new problem on the other 40G member of that same UM-MSU link. This seems to be resolved after resetting one transponder at the UM/Merit router interface.

         

        Server room:

        working on replacing the PDUs for some racks at UM site

        Planning on replacing the WAN (replacing all WAN switches and reposition and recable machines) of UM Site. Placed order for Dell switches (9 S5232 and 9 S3048). Plan a week of downtime, maybe align with the moving time of MSU site(March or April), to rearrange the server room. 

         

         
         
      • 13:45
        MWT2 5m
        Speakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))

        UC

        • Were seeing 3-4k open connections to the AGLT2 xcache a couple of weeks ago. After Ilija restarted xcache,  things seem to be behaving better
        • Applying necessary configuration changes to UC workers for migrating to SLATE frontier-squids
        • UChicago SciDMZ egress maxed out 01 Feb

        IU

        • New head node equipment received, in the process of being built
        • PO for new network equipment submitted

        UIUC

        • Transceivers ordered for the new compute nodes
      • 13:50
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))

         

        We recovered from 1 week downtime of NET2_DATADISK, _SCRATCHDISK

        Extensive & useful discussion at the ADC weekly yesterday.  Multiple followups likely.   

        During the downtime, we successfully ran using just NESE Ceph storage.

        Lincoln and Ilija are helping us run squids on our SLATE node (re: low priority DDM ticket).

         

         

      • 13:55
        SWT2 5m
        Speakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))

        OSG and XRootD software updates progressing:
            -- working on these in our test cluster
            -- lost a couple of days - updating our local Tier-3 cluster, now done

        New perfSONAR nodes have been ordered - awaiting delivery

        Met with campus networking personnel to discuss IPv6 implementation:
            -- they are making progress
            -- a few issues they still need to address (DNS, etc.)
            -- decision about the current switch in the data center that provides outside connectivity
            -- we're setting up a host for testing purposes 

         

        OU:

        -- site running well, getting good 8-core opportunistic throughput as well

        -- waiting on Dell quotes for new compute node purchases

         

    • 14:00 → 14:05
      WBS 2.3.3 HPC Operations 5m
      Speakers: Doug Benjamin (Duke University (US)), Lincoln Bryant (University of Chicago (US)), lincoln bryant

      NERSC -

      Smooth running - getting lots of cycles see attached plot

      Theta -

      getting hours - see attached plot. Now have up to 10 jobs (384 nodes each for 9 hours) in the system.

      If we don't use alot of hours in the next two weeks we will lose them due to ALCC pullback. US CMS is not running anything. so usage has to be all ATLAS.

       

      Frontera:

        - Occasional issues with $WORK (lustre) filesystem. Harvester cacher data nominally belongs here, but occasional "input/output" errors on this device. Files become non-deleteable by normal means (rm, find -delete ..) and Harvester cannot continue. Must file a ticket to get system admins to delete the file. This paused jobs for the last week or so as Lincoln didn't catch that this error had returned until yesterday. Will consider moving Harvester files back to $HOME or possibly $SCRATCH if issues continue. 

       - System now in downtime through the 8th for "texascale" (large scale) mode. 

    • 14:05 → 14:20
      WBS 2.3.4 Analysis Facilities
      Convener: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:05
        Analysis Facilities - BNL 5m
        Speaker: William Strecker-Kellogg (Brookhaven National Lab)
      • 14:10
        Analysis Facilities - SLAC 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:15
        Analysis Facilities - Chicago 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

        All working fine.

    • 14:20 → 14:40
      WBS 2.3.5 Continuous Operations
      Convener: Ofer Rind
      • AGIS to CRIC migration proceeding again, see Alexey's talk at Monday's TCB meeting
      • Some residual operational issues during/subsequent to NET2 GPFS intervention
      • Robust site topic discussion at the ADC weekly (US and UK)
        • Discussion led to ideas for reconfiguration of PQs and SEs at NET2/NESE
      • SLATE Frontier-squid deployment proceeding at NET2 (GGUS)
      • 14:20
        US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5m
        Speakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
      • 14:25
        Service Development & Deployment 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

        Xcaches working fine. Updated to v5.1.0rc5. 

        VP working fine. Issues with getting vps.cern.ch externally visible loadbalancer to work correctly.

        Squid deployments are being worked on at MWT2 and NET2.

      • 14:30
        Kubernetes R&D at UTA 5m
        Speaker: Armen Vartapetian (University of Texas at Arlington (US))

        The computers I am using I borrowed from our admins, are several test machines usually used for evaluation of updates and services before they are implemented in the main SWT2 grid production cluster.

        Right now the test cluster is utilized for such evaluation, and as part of it is undergoing complete rebuild, with possible hardware changes. Once I get it back in 1-2 week, I'll reinstall k8s, and proceed with setting up k8s test queue.

    • 14:40 → 14:45
      AOB 5m

      This afternoon is the monthly ESnet Blueprinting meeting for those who want to join.

      Meeting Notes: https://docs.google.com/document/d/1zng0gXq-OsPyw0km0JygVDQM0ZQK806jXOQcACTURlQ/edit

      Eli Dart is inviting you to a scheduled Zoom meeting.

      Topic: Eli Dart's Zoom Meeting
      Time: Feb 3, 2021 11:30 AM Pacific Time (US and Canada)

      Join Zoom Meeting
      https://esnet.zoom.us/my/elidart?pwd=dmx6TGZEalBENmN1L3ZVVkllZy9NUT09