US ATLAS Computing Facility

→ US/Eastern
Description

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  996 1094 4232

Meeting password: 125

Invite link:  https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09

 

 

    • 13:00 → 13:10
      WBS 2.3 Facility Management News 10m
      Speakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)

      1) thanks for the QR contributions.  Did we miss anything?

      2) Any updates to capacity spreadsheet pending?

      3) future analysis systems facilities workshop this week will inform some of our activities going forward.

      • Firstly, REANA deployment at BNL.  Use k8s at BNL and containerized deployment.  There is some dev work here to do, c.f. https://github.com/orgs/reanahub/projects/33
        • Eric reports on-going work with REANA-slurm
        • Eric also reports availability of OpenShift (Enterprise RedHat Kubernetes) they have at BNL that could be used for protopyping
        • Kasuhik notes we need buy-in from Varena, WBS 5, to support users when it becomes production ready.
      • Independently on IRIS-HEP SSL we'll explore deploying JupterLab, JupyterHub -to- Dask scheduling over HTCondor, deployed from Kubernetes.
        • This is most similar to the UNL's AF (c.f. this talk and others)
        • We may need to explore an Authz service compliant with CERN OIDC and e-groups, eventually.  
          • (This was suggested by Brian B after comments at the blueprint, to be investigated; a longer term project project - n.b. we already have CILogon-to-CERN OIDC for identity, but not the authorization part.)
        • A related related IRIS-HEP SSL activity is to build a federated Kubernetes (multi-cluster) with a number of contributing sites (UC, UNL, UW, UCSD). The second meeting of this group is next Tuesday.
      • Likely management will want to weigh-in, so to be discussed.

      5) Second k8s-hep meetup is Dec 1,2 https://indico.cern.ch/event/968726/ - a technical meeting for sharing Kubernetes knowledge in the HEP context. 

      6) We need to ramp up the devOps team for frontier-squid now that we have a slate-registered Kubernetes cluster at each Tier2 and Frontera:

      7) Data lakes meeting again this afternoon (Shawn organizing)  Meeting notes at https://docs.google.com/document/d/1HII7CpcIJP3uyhlZahLt6L2qI7dG_JDkMBWa7OpQ2GE/edit

    • 13:10 → 13:20
      OSG-LHC 10m
      Speakers: Brian Lin (University of Wisconsin), Matyas Selmeci

      VO client update (tomorrow)

      • Fix data for the WLCG VO due to upstream CA changes

      OSG 3.5.26 (likely next week)

      • HTCondor 8.8.11
      • CVMFS 2.7.5
      • scitokens-credmon 0.8.1
      • python-scitokens 1.2.4 (Python 3 support)
      • Upcoming: HTCondor 8.9.9, XRootD 5.0.2
    • 13:20 → 13:35
      Topical Reports
      Convener: Robert William Gardner Jr (University of Chicago (US))
      • 13:20
        TBD 15m
    • 13:35 → 13:40
      WBS 2.3.1 Tier1 Center 5m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
      • On 10/22, a configuration change in dCache (combine DMZ read and write pools into readwrite pool) caused transfer failures and BNL PQ blacklisted (stage-in errors). Rolled back, issue solved.
      • MAS R&D : moving more DAODs to LAKE
      • updated T1 numbers on the capacities spreadsheet
    • 13:40 → 14:00
      WBS 2.3.2 Tier2 Centers

      Updates on US Tier-2 centers

      Convener: Fred Luehring (Indiana University (US))
      • Good running with very few tickets/outages over the past 2 weeks.
        • No need for plots this week.
      • I would like everyone to clean up and remove unneeded panda queues and decommissiond  clusters. In validating the accounting data, I got confused by 2 similarly named clusters one of which was no longer in service,
      • I find the hodge podge of site names confusing so I made a strawman proposal for naming the sites so they show up together in CRIC and Panda Monitor to a few of you:
        • US-ATLAS-AGLT2
        • US-ATLAS-T1
        • US-ATLAS-MWT2
        • US-ATLAS-NET2
        • US-ATLAS-SWT2-OU
        • US-ATLAS-SWT2-CPB
        • US-ATLAS-SWT2-UTA
      • We have been asked by Julia Andreeva to make a presentation on November 22 at the WLCG/HSF "...to give a presentation summarizing US ATLAS T2 experience or probably representatives of a particular site would like to share their experience/plans?"
      • 13:40
        AGLT2 5m
        Speakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)

            No ticket

            - had one file access problem flagged to us last Friday, but files exist, believed resolved.

            Hardware

            - 3 new VMware nodes for UM
              Powerful, 512G RAM, 256 HT, 6.4TB NVME, 25G NIC
              Racked and ESXi installed.  
              Need further cabling and configuration to be put into production.

            - MSU new storage online (472 TB usable)
              retired 2 TB disks from ~2010 (300 TB)
              currently migrating data for retiring one MD3060e with history of problems
              will re-locate the 60x 3 TB disks from the MD3060e
              
           Services

           - Scheduling OS upgrade our main switches (Juniper EX9208) at both sites
             Next Tuesday 03-Nov-2020
             Should only involve 2-3 ~15mn reboots at both sites and no loss of jobs
             will not set site offline, just at risk: intermittent outage

           - Adding anti-spoofing filters to our main switches
             to avoid abuse of our DNS servers
             by spoofing source address to come from our subnets

         

      • 13:45
        MWT2 5m
        Speakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))

        UC

        • New workers in production. Accounting updated for both UC and IU
        • One of the storage nodes crashed last Wednesday
        • Investigating GGUS ticket

        IU

        • Continuing to work with the IU NOC for debugging

        UIUC

        • Purchase order submitted, still waiting on delivery details
      • 13:50
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))

        Smooth operations with a full site for the past two weeks.

        SLATE xCache working.  

        Restarted space token dumps at the request of ADC.

        One of the nics on a GPFS NSD was failing and needed to be disabled.  

        Usual low level of old CPUs failing with temperature alarms or failing disks. 

        Lots of NESE-related planning happening. 

        On our agenda:  OSG 3.5, HTTPS-TPC enabling, ipv6

      • 13:55
        SWT2 5m
        Speakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))

        OU:

        - Nothing to report, all running well.

         

    • 14:00 → 14:05
      WBS 2.3.3 HPC Operations 5m
      Speakers: Doug Benjamin (Duke University (US)), lincoln bryant

      debugging going slowly for ALCF Theta. Task migrated from NERSC is not starting. (gets killed by Harvester)

      Networking issues at TACC sorted out and testing to run multiple pilots per compute node resumes

      NERSC ran out of Inode quota causing Sample A jobs to fail. fixed and work is resuming.  got another 5M hours for NERSC. 

      BNLHPC_DATADISK storage is full.  Files need to be removed from it. 

    • 14:05 → 14:20
      WBS 2.3.4 Analysis Facilities
      Convener: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:05
        Analysis Facilities - BNL 5m
        Speaker: William Strecker-Kellogg (Brookhaven National Lab)
      • 14:10
        Analysis Facilities - SLAC 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:15
        Analysis Facilities - Chicago 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

        All working fine. Need to restart FoldingAtHome.

    • 14:20 → 14:40
      WBS 2.3.5 Continuous Operations
      Convener: Ofer Rind
      • 14:20
        US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5m
        Speakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
      • 14:25
        Service Development & Deployment 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

        Analytics

        • ES upgraded 7.9 - working on Auditing
        • Wrote deduplicator for perfSONAR.
        • Working on Spark K8s deployment at CERN. Needed for DDM and popularity studies

        XCache and VP

        • SLATE and XCache working fine at all US sites. 
        • LRZ VP queue is back on - need to scale them back up.
        • I still need to create an extra ATLAS site for ANALY_BNL_VP queue. Once done we will try to scale it up.
    • 14:40 → 14:45
      AOB 5m