US ATLAS Computing Facility

→ US/Eastern
Description

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  996 1094 4232

Meeting password: 125

Invite link:  https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09

 

 

    • 13:00 → 13:10
      WBS 2.3 Facility Management News 10m
      Speakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)

      Just a heads up and invitation to attend next week's k8s-hep meetup, https://indico.cern.ch/event/968726/.    Next week's WBS 2.3 Coordination meeting will be cancelled.

    • 13:10 → 13:20
      OSG-LHC 10m
      Speakers: Brian Lin (University of Wisconsin), Matyas Selmeci

      - XRootD 4.12.5 (for OSG 3.5 series) still in testing; no complaints yet

      - XRootD devs say 5.1.0 will be out soon; if so, will skip 5.0.3 and go directly from 5.0.2 to 5.1.0 (for upcoming repos)

      - HTCondor-CE, Gratia probe transition to EL 8 / Python 3 making good progress

      - Expecting HTCondor 8.9.10 release next week (for upcoming repos)

       

      Notes from Ofer:

      • xrootd upgrade needd for http-tpc activation
      • issue was whether we'd have a backport back to osg 3.5
      • what is the deadline anyway? Alessandra: May.
      • Feeling was we could get it done by May w/ 3.6 upgrade, which is sched for Feb.
      • But concerns are heavy upgrade - so sites may not want to update right away
      • So we still have quetions to work out - we need Brian

      Wei - challenge in OSG 3.6 globus support gets dropped.  So concerns that everything may not work out of the box. Would like OSG to support xrd 5 in osg 3.5.  Both Ofer and Wei feel strongly about it.   

      Doug - claims you still need gridftp to still be there.

      Can sites go to xrd 5 w/o osg 3.5; q for swt2 and net2. 

       

       

       

    • 13:20 → 13:35
      Topical Reports
      Convener: Robert William Gardner Jr (University of Chicago (US))
      • 13:20
        New US LHC Analysis Facilities Forum 15m
        Speakers: Doug Benjamin (Duke University (US)), Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Kaushik De (University of Texas at Arlington (US)), Mark Neubauer (Univ. Illinois at Urbana Champaign (US)), Mike Hance (UC Santa Cruz), Paolo Calafiura (Lawrence Berkeley National Lab. (US)), Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Verena Ingrid Martinez Outschoorn (University of Massachusetts (US)), Wei Yang (SLAC National Accelerator Laboratory (US)), William Edward Strecker-Kellogg (Brookhaven National Laboratory (US)), William Strecker-Kellogg (Brookhaven National Lab)
    • 13:35 → 13:40
      WBS 2.3.1 Tier1 Center 5m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))

      Normal operations in general. 

      REMINDER: Network intervention December 1st 6am-6pm.

      Preliminary work on cost evolution and estimation of disk vs tape presented at the WLCG this week. Bandwidth requirements for HL-LHC make tape systems challenging. Work will continue, sharing infos. with FNAL.

      Happy Thanksgiving! 

       
    • 13:40 → 14:00
      WBS 2.3.2 Tier2 Centers

      Updates on US Tier-2 centers

      Convener: Fred Luehring (Indiana University (US))
      • Good running with low error rates and very few GGUS tickets. For the last ~1 week here are the numbers of jobs finishing in the various possible states:
      Site Done Failed Canceled Closed %Failed
      AGLT2 (no BOINC) 107160 3060 96 288 2%
      MWT2 241204 8736 197 306 3%
      NET2 93148 9053 24 5 8%
      OU_ATLAS 3492 25 2 0 0%
      OU_ATLAS_OPP 4289 23 4 89 0%
      SWT2_CPB 88990 9679 40 0 2%
      UTA_SWT2 21343 35 7 0 0%
      •  Could everyone please report on how they are doing with their infrastructure/networking purchases.
      • Already covered OSG 3.5.
      • 13:40
        AGLT2 5m
        Speakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)

        One Ticket :
           GGUS ticket 149378: morphed into 2 issues.
           - originally poor (~zero) transfer rate to UK
           - recently : One file missing from our dCache.
             Traced to day and pool with hardware failure.
             Declared a list of files lost files but that file was missing.

        Services :

           dCache updated from 6.2.5 to 6.2.10. Includes security fix.  Smooth upgrade.  
           
           New AMD VMware cluster at UM now working well after simplifying network connections,
           but with downside of more limited path redundancy.  Had VMs in distributed vswitch losing connectivity.
           Could recover by migrating to different host.  Seemed to be tied to mixing uplink switches.

           New concern: Newer VMware versions not supporting Dell MD storage (iscsi at UM / direct attach at MSU)
           current VMware version 6 seems to work, but version 7 may not... or cause support issues.
           We are investigating new storage options.

        Hardware:

           MSU retired 300 TB from last 2TB disks.

           UM retired one problematic MD3060e storage enclosure (3TB*60 disks).
           The controllers, power supplies, drawers can become spares.
           Will move 60*3TB disks to recently freed MSU MD1200s to maitain storage capacity.

           UM old VMware cluster and old perfsonar nodes rebuilt into work nodes, adding 352 core.

           MSU last purchase of C6420s in production (later today), adding 576 cores

      • 13:45
        MWT2 5m
        Speakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))

        Debugging 1099 errors. Found that, for the compute nodes using XL710 chip and i40e driver on 10Gbps, upgrading the kernel from the stock 3.x series to the 5.x ml series, errors appear to have gone away for the past ~24 hours. Will upgrade 8 workers at IU and update to kernel-ml today and let run over the holiday weekend.

        dCache upgrade scheduled for this coming Monday

        Still investigating HTTP-TPC issues. It appears that the CRLs retrieved by running fetch-crl directly on the dCache machines differ for a couple of CAs in OASIS. Will move all dCache machines to use locally-installed CAs during Monday's downtime

        New UIUC nodes expected to arrive next week

      • 13:50
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))

         

        Smooth operations, site full, no tickets.

        Working on HTTP-TPC via xrootd.  

        Preparing to upgrade to OSG 3.5.

        Progress on ipv6.

        Lots of NESE prep happening, talk at WLCG workshop. 

      • 13:55
        SWT2 5m
        Speakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))

        UTA:

        1) All storage from most recent purchases is up, but putting the last few systems into production on hold for:

            (i) Retirement decision about some MD3660i vintage storage

        2) Resurrected 4 of 6 problematic compute nodes. Remaining 2 will require on-site support

        3) Generally smooth operations

         

        OU:

        - Nothing to report, all running well

        - Will probably have OSCER maintenance downtime Wed, Dec 16

         

    • 14:00 → 14:05
      WBS 2.3.3 HPC Operations 5m
      Speakers: Doug Benjamin (Duke University (US)), lincoln bryant
    • 14:05 → 14:20
      WBS 2.3.4 Analysis Facilities
      Convener: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:05
        Analysis Facilities - BNL 5m
        Speaker: William Strecker-Kellogg (Brookhaven National Lab)

        upcoming downtime (all of 12/1) will affect tier 3 as well. announcements have been made

      • 14:10
        Analysis Facilities - SLAC 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:15
        Analysis Facilities - Chicago 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

        Everything works fine.

        Frontier platform needed quite a bit of updates. Now almost all of works. Working with Nurcan and Andrea on getting it fully out of Millisa's space.

        New data source will be coming to ES: Each job perfmon data will be sent by pilot to UC logstash, indexed and analyzed in atlas-ml.org.

         

    • 14:20 → 14:40
      WBS 2.3.5 Continuous Operations
      Convener: Ofer Rind
      • AGS to CRIC Migration:  Editing DDM endpoints now CRIC only.  Further migration over the next two weeks.
      • 14:20
        US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5m
        Speakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
      • 14:25
        Service Development & Deployment 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

        XCache  

           creating special image to test issues with TLS between different versions of xroot servers.

           creating a new 5.1.0 image to test new g-Stream.

        VP

           running at: BNL, MWT2, AGLT2, SWT2, NET2, LRZ-LMU, Prague2

           making a new Prague_VP site as a diskless one. Waiting on Petr. 

          Performance is good and getting better.

          Issues:

             NET2 does not work as origin as it does not expose xrootd door

             issue reading from SLAC

           Need some developments that need to be agreed with Rucio people.

    • 14:40 → 14:45
      AOB 5m