US ATLAS Computing Facility

→ US/Eastern
Description

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  996 1094 4232

Meeting password: 125

Invite link:  https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09

 

 

    • 13:00 → 13:10
      WBS 2.3 Facility Management News 10m
      Speakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
    • 13:10 → 13:20
      OSG-LHC 10m
      Speakers: Brian Lin (University of Wisconsin), Matyas Selmeci

      OSG 3.5.34

      • Python 3 Gratia probes
      • CVMFS 2.8.1 (also in OSG 3.6)

      Impending updates

      • HTCondor 8.8.13 in OSG 3.5 release
      • HTCondor 9.0.0 in OSG 3.5 upcoming and OSG 3.6
      • HTCondor-CE 5.1.0 in OSG 3.5 upcoming and OSG 3.6
      • XRootD 5.2.0 (or 5.1.2?) in OSG 3.5 upcoming and OSG 3.6
      • XRootD, HTCondor, and HTCondor-CE in EL8

      HTCondor sites should register for HTCondor week! https://agenda.hep.wisc.edu/event/1579/

    • 13:20 → 13:35
      Topical Reports
      Convener: Robert William Gardner Jr (University of Chicago (US))
      • 13:20
        TBD 15m
    • 13:35 → 13:40
      WBS 2.3.1 Tier1 Center 5m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
    • 13:40 → 14:00
      WBS 2.3.2 Tier2 Centers

      Updates on US Tier-2 centers

      Convener: Fred Luehring (Indiana University (US))
      • Relatively tough period with various problems - I ask each site about drops in the number of jobs profile.


      • Quarterly reporting due!!!
        • Shawn wants my report on Friday so please enter your information by the end of Thursday.
        • Update the capacity sheet (https://bit.ly/usatlas-capacity) as of 3/31 with purchases and retirements.
        • Update the services sheet (https://tinyurl.com/yytakxdr) as of 3/31 with any new versions of middleware, storage system software etc.
      • Still not able to implement http-tpc at xrootd sites. Still asking all sites to jump to OSG 3.5
      • Need to check the March accounting numbers. Wenjing has agreed to help. After a revision ~1 week ago the number are looking much more reasonable.
      • Big discussion at the Monday ADC Technical Coordination meeting about IAM / Tokens. ADC was not planning to switch to tokens before the end of Run 3 but it seems Globus has forced their hand.
      • What about IPV6?
      • 13:40
        AGLT2 5m
        Speakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)

        1). working on provisioning with the new switches.

        2). GGUS ticket about dCache transfer error caused by fiber break of merit network starting from 9am 12nd Apri, the provider is still working on the repair. 

        3) Had over 1000 production jobs overusing the memory, either killed by pilot or system oom killing.

        4) had high memory jobs (>6G) flood (>4000 cores), we put a 1500 core limit on the CRIC PanDA queue, and also modified the gatekeeper routing rules, to make sure high memory jobs only go to the big memory work nodes.

        5) rebuild most of the work nodes with dedicated partition for BOINC jobs (which originally use /tmp as work directory) to avoid interference with system and condor jobs

        6) received 4 recently purchased lenovo/forsa work nodes, working on provision

      • 13:45
        MWT2 5m
        Speakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))

        WebDAV re-enabled as primary protocol for MWT2

        Partial dCache outage this morning due to one of the MD3460s going offline again. David is working on bringing the storage back online

        IU networking upgrade ongoing

      • 13:50
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))

         

        1. Rolling upgrade to centos 7.9 almost done.  

        2. Smooth operations otherwise modulo one dip in production seemingly for just lack of panda jobs. 

        3. Preparing to purchase 2-4 gateway nodes for NESE Ceph storage.

        4. NESE core networking will be upgraded on April 22.  NESE_DATADISK,_SCRATCHDISK will be down.

        5. We will have a 5 minute network outage tomorrow after 5 to set up ipv6 BGP peering between BU and ESNet.

        6. We have a container with Xrootd 5.1.1, successfully exporting GPFS.  Testing now, plan to jump to 5.2(?) 5.1.2(?) when it appears in OSG. 

      • 13:55
        SWT2 5m
        Speakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))

        OU:

        - Nothing to report, all running well.

        UTA:

        • Remapped OSG resource SWT2_CPB_SE
          • Changed from SRM to GridFTP service
          • created one hour downtime in YAML; successfully blacklisted and recovered RSE's
          • Blunt instrument; but works
        • IPV6 progressing
          • Hopefully first testing this week

         

    • 14:00 → 14:05
      WBS 2.3.3 HPC Operations 5m
      Speakers: Doug Benjamin (Duke University (US)), lincoln bryant

      Very good running at NERSC. Increased number of running workers from 5 to 8.

      Each worker (batch submission) has 200 KNL nodes. 6 hours per submission.

      Lincoln is running test jobs on his test instance to make sure he can operate the production instance using his credentials  (grid and globus).

      No running at TACC - Rob submitted revised proposal for more time at TACC.

       

    • 14:05 → 14:20
      WBS 2.3.4 Analysis Facilities
      Convener: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:05
        Analysis Facilities - BNL 5m
        Speaker: William Strecker-Kellogg (Brookhaven National Lab)

        Normal operations, awaiting EOS client upgrade with KRB5 env. var fix

      • 14:10
        Analysis Facilities - SLAC 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:15
        Analysis Facilities - Chicago 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
    • 14:20 → 14:40
      WBS 2.3.5 Continuous Operations
      Convener: Ofer Rind
      • Changing protocol from SRMV2 to gridftp in OSG topology seemed to fix downtime declaration problem for SWT2
        • Reviewing site topologies to identify remaining SRMV2 protocol settings to change
        • Site admins please notify Mark of any downtime issues encountered relating to CRIC, OSG topology, Hammercloud, Switcher, etc....we are trying to compile and document the problems
      • BNL now fully webdav for WAN transfers
        • See attachment for list of remaining sites still on SRM/gsiftp (Alessandra)
        • Need a plan for Tier-3s - xrootd seems viable, little use of http
      • SWT2 and BU site plans to move to OSG 3.5?
        • SWT2 next week?
        • OU as soon as XRootd 5.1.2 (or 5.2) released and stable; gatekeeper will be updated when new hardware arrives, expected in "a month or two"
        • BU in process, testing 5.1.1
      • Frontier-Squid DevOps meeting today
        • Enabling k8s writing of logs to standard out caused pod flapping at MWT2 - need to turn off for stable running; continuing to investigate cache hit rate, will know more next week
      • Quarterly reports
      • 14:20
        US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5m
        Speakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
      • 14:25
        Service Development & Deployment 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
        • VP cache target now identified within Rucio in latest release
          • Xcache servers now must provide heartbeats in order to receive data
    • 14:40 → 14:45
      AOB 5m