US ATLAS Computing Facility

→ US/Eastern
Description

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  996 1094 4232

Meeting password: 125

Invite link:  https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09

 

 

    • 13:00 → 13:10
      WBS 2.3 Facility Management News 10m
      Speakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn Mc Kee (University of Michigan (US))
    • 13:10 → 13:20
      OSG-LHC 10m
      Speakers: Brian Hua Lin (University of Wisconsin), Matyas Selmeci

      Release (this week)

      • XRootD 5.5.0
      • osg-token-renewer permission bug fix

      Miscellaneous

      • Is there any US ATLAS interest in EL9?
      • Mat, BrianB, and I are planning to be at the IRIS-HEP planning retreat
      • Please update to HTCondor-CE 5.1.5
    • 13:20 → 13:50
      Topical Reports
      Convener: Robert William Gardner Jr (University of Chicago (US))
    • 13:50 → 13:55
      WBS 2.3.1 Tier1 Center 5m
      Speakers: Doug Benjamin (Brookhaven National Laboratory (US)), Eric Christian Lancon (Brookhaven National Laboratory (US))

      Steady as she goes. 

    • 13:55 → 14:15
      WBS 2.3.2 Tier2 Centers

      Updates on US Tier-2 centers

      Convener: Fred Luehring (Indiana University (US))
      •  Reasonably good running over the last month.
        • NET2 was shutoff and than turned back on by ADC.
        •  Short issue at CERN on September 8. (Rucio certificate expired.)
      • Tier 2 milestone meeting on this Friday at 4 pm CDT.
      • 13:55
        AGLT2 5m
        Speakers: Philippe Laurens (Michigan State University (US)), Dr Shawn Mc Kee (University of Michigan (US)), Prof. Wenjing Dronen

        No significant problem 


        A couple of dCache files hang at access, 
        all from one pool, msufs13_1, 
        restarting the pool resolved the issue. 

         

        Upgraded condor to 9.0.16, No problem.

        condor-CE gate01/03 updated too

        $HTCondorCEVersion: 5.1.5 $ 

      • 14:00
        MWT2 5m
        Speakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))

        Overall running well.

        Planning to update condor on our central managers in the next couple of weeks.

        Discussing upcoming purchasing.

      • 14:05
        NET2 5m
        Speakers: Prof. Rafael Lopes de Sa (University of Massachussets, Amherst), William Axel Leight (University of Massachusetts Amherst)

        NET2.0: Problems with transfer to NESE. Augustine Abaris from BU investigating NESE DTN. Problem persists but it is mitigated after reducing the number of jobs running.

         

        NET2.1: Rack 1 space at MGHPCC freed up. Working on ESnet connection. Best strategy for this year is to do 10GB/s via NoX. We are trying to plan for 100 GB/s (with equipment capable of 400GB/s) for next year via NEREN. Partial inventory of NET2.0 equipment made with help of SBU, hoping to complete the full inventory with help from BU and UMass RC teams. Some glaring discrepancies with US-ATLAS facilities spreadsheet that we are trying to understand. Discussions about NESE EC configuration on-going on MM.

      • 14:10
        SWT2 5m
        Speakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))

        OU:

        • Running well in general.
        • There was a network issue at OneNet in Tulsa which impacted inbound transfers to OU last week. Fixed.
        • This caused temporary draining of site.

        UTA:

        • Ongoing work in K8s, job mix balance, Need to look at accounting for APEL
        • Problems with trying to create new SLATE development host; OSG is working on it
        • Cluster running smoothly
    • 14:15 → 14:20
      WBS 2.3.3 HPC Operations 5m
      Speakers: Lincoln Bryant (University of Chicago (US)), Rui Wang (Argonne National Laboratory (US))

      Data transfer issues resolved. Jobs are now completing from Cori.

       

      Rui working on TACC proposal, Lincoln assisting for parts.

       

      Perlmutter scratch filesystem still under maintenance.

    • 14:20 → 14:35
      WBS 2.3.4 Analysis Facilities
      Conveners: Ofer Rind (Brookhaven National Laboratory), Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:20
        Analysis Facilities - BNL 5m
        Speaker: Ofer Rind (Brookhaven National Laboratory)
      • 14:25
        Analysis Facilities - SLAC 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:30
        Analysis Facilities - Chicago 5m
        Speakers: Fengping Hu (University of Chicago (US)), Ilija Vukotic (University of Chicago (US))
        • we improved our jupyterlab interface for GPU selections. Previously users were given GPU selection by memory size but can't see the availability information, so notebooks can endup in waiting for a very long time.
        • added cvmfs health monitoring in troubleshooting intermitant cvmfs availability issue on htcondor pods
        • we are helping two demonstrations for PyHEP (one was this morning, next tomorrow morning)
    • 14:35 → 14:55
      WBS 2.3.5 Continuous Operations
      Convener: Ofer Rind (Brookhaven National Laboratory)
      • NET2.1 Migration discussions on Mattermost (Ceph and Infrastructure)
        • Augustine working on NESE storage server load issues
      • XCache prefixing issue in Rucio, status?
      • Interesting talk by Rod Walker at this morning's GDB meeting about preparing for and dealing with expected spike in European energy costs, likely starting in January
      • Federated DevOps meeting: Notes  
      • 14:35
        US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5m
        Speaker: Mark Sosebee (University of Texas at Arlington (US))
      • 14:40
        Service Development & Deployment 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
        • XCache
          • all running fine
          • will be testing v5.5 next week
        • VP
          • Rucio issue made xcache paths not prepended thus disabling all xcaches. VP queues still worked but read directly from origins.
          • Fixed last night and now things are back to normal.
        • Analytics
          • Now we have all of the Squids monitored in UC ES.
          • Today we will have all of the xAOD access snooping data indexed back in ES. All 7 years of data. 
        • ServiceX
          • Today we should have v1.0.31
          • Will install have it installed in preproduction today and if everything is fine on AF on Monday.
        •  
      • 14:45
        Kubernetes R&D at UTA 5m
        Speaker: Armen Vartapetian (University of Texas at Arlington (US))
        • The K8S cluster was stable past weeks running production jobs.
        • One change in the configuration in CRIC was to limit SCORE jobs, after SCORE jobs gradually occupied most of the cores. After the limit was in place we were back to reasonable mixture of SCORE/MCORE jobs. To understand how to optimize the limits and maybe the method itself.
        • Monday/Tuesday we had some nodes in stuck state (didn't happen before). Looks like some nodes run out of memory. Just restarting helped, and today all looks fine, but this needs some further investigation to understand the root cause.
    • 14:55 → 15:05
      AOB 10m