US ATLAS Computing Facility

→ US/Eastern
Description

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  996 1094 4232

Meeting password: 125

Invite link:  https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09

 

 

    • 13:00 → 13:10
      WBS 2.3 Facility Management News 10m
      Speakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)

      Yesterday we had our third meeting to discuss USATLAS planning for Data Lakes / QoS.  Meeting notes at https://docs.google.com/document/d/1h5FNHuSA23k1tdQxRZSFIn3AutEv8V98OtkogWm45Pc/edit while our long term planning document is available at https://docs.google.com/document/d/1PwpQpgD1xLpvqewmLW9xfgWVOOANFJlkCfcons74Mms/edit#heading=h.9v3o1er2wax7

      • The meeting had a lot of discussion and the main action item is to describe two possible models, incorporating the monitoring and simulation results we have so far.   Currently two models are proposed:  1) Ilija's model where all PRIMARY data is on tape and caches are deployed in front of tape and at sites and 2) Saul's model trying to utilize what commercial clouds have been doing (MinIO, networks, cloud services).
    • 13:10 → 13:20
      OSG-LHC 10m
      Speakers: Brian Lin (University of Wisconsin), Matyas Selmeci

      OSG 3.5.27/3.4.57 (this week)

      • osg-wn-client installs are failing due to a newly released xrootd4 package in EPEL. New gfal2 build is in osg-testing for 3.4/3.5 that fixes this issue. In the meatime, the following workaround can be used: yum install --exclude xrootd4 osg-wn-client
      • HTCondor 8.9.9 (upcoming)
      • XRootD 4.12.5 (3.5 only)

      Other

      • EL8: XRootD and HTCondor are nearly ready; CE packages are still under dev
      • osg-software, osg-int, and vdt-discuss mailing lists are going away (replaced by software-discuss and site-announce) Nov 20. Subscribers are not being copied over from the old lists. Expect an email with subscription details this week.

       

    • 13:20 → 13:35
      Topical Reports
      Convener: Robert William Gardner Jr (University of Chicago (US))
      • 13:20
        TBD 15m
    • 13:35 → 13:40
      WBS 2.3.1 Tier1 Center 5m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
    • 13:40 → 14:00
      WBS 2.3.2 Tier2 Centers

      Updates on US Tier-2 centers

      Convener: Fred Luehring (Indiana University (US))
      • Smooth running in the last weeks - no outages, few tickets.
      • I am trying to close last quarter's capacity numbers. I am chasing the sites one by one to confirm that they have the correct numbers for September 30 to be put on the v53 sheet:
        • https://docs.google.com/spreadsheets/d/1nZnL1kE_XCzQ2-PFpVk_8DheUqX2ZjETaUD9ynqlKs4
        • Please check the CPU numbers and the Storage numbers (column L).
        • Also please check the facilities services information on that sheet: https://docs.google.com/spreadsheets/d/1_fKB6GckfODTzEvOgRJu9sazxICM_RN95y039DZHF7U/
        • v53 will be completely locked soon probably later today
        • v54 for the quarter ending on December 31 is open
      • Lincoln spotted a real issue with the pilot where the pilot itself ended up with 5 GB of memory causing memory exhaustion.
        • Paul Nilsson found that a logging service was leaking memory in some cases. Rather than fix the service, it was possible to stop using it. There should be a new pilot version later this week.
      • How are we doing on moving to OSG V3.5?
      • Need to check the October accounting numbers. September's numbers were ~correct.
      • Every site should replace their head nodes / gate keepers / control nodes / etc. that are out of  warranty.
        • We don't want down time caused by failure of this type of server.
        • I will be asking you how old your servers are.
      • Having a small meeting with Mayuko, myself, Shawn,  Mark, Ofer, and Judith to start to understand stuck transfers.
      • 13:40
        AGLT2 5m
        Speakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)

        1 ongoing ticket 149378: a big percentage transfer failure . We found one of the dcache door nodes (dcdum01) having errors about connecting to the head node, we rebooted the node and it fixed the problem. 

        2 solved ticket:

        149268 : files offline during transfer, this is because one dcache pool msufs04_2 became offline, and it was a quick fix after we restarted the pool. 

        149328: no transfers for several hours.  Our dcache head node (hosting pnfs and chimera database) died (raid controller failure) due to room overheating (machines shut themselves down). We had to migrate the system disks and nvme disks (hosting the database data) to another machine, and bring that machine up as the new head node. 

        Software updates:

        We did (both sites) update/catchup the OS+FW on our core EX9208 Juniper routers. The update at MSU had to be partially rolled back because of bug causing instability. After the update of routers, the heat load observed is higher in most blades. Most MSU switch blades returned to normal temperature after partial OS roll back. 

        We purchased and built 3 new Dell R7525 machines for the UM vmware cluster. We have migrated all vm instances from the old R740 Intel cluster to the new R7525 AMD cluster. We experienced a lot of network issues (most vms have partial network connectivity and require migrating to another host to resolve the problem)during the migration and we are identifying/resolving it gradually. 

        OS:

        In the process of upgrading more SL6 servers to CC7, already finished dns server upgrade. 

        Hardware:

        Replaced one more sick PDU in MSU T2

        At MSU T2. Noticed dust buildup on the EX9208 air filter and vacuumed from outside.

        At UM T2, we hired a company to clean the dust in the server room, under the rack, outside of the machines. 

         
      • 13:45
        MWT2 5m
        Speakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))

        One of the md3460 dCache servers crashed again yesterday evening. Back online as read-only for now while we troubleshoot

        Investigating pilots that are using a lot of memory and are OOMing workers

        Ongoing 1099 error debugging. We have been able to replicate timeouts in `rucio download` from a worker (iut2-c285)

        Continuing to debug poor IU IPv6 performance 

        Investigating TPC HTTP smoke test issue with GermanGrid CRL

      • 13:50
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))

         

        Smooth operations except for one black hole worker node and a several hour DDM outage due to a power issue at MGHPCC affecting NoX networking equipment. 

        BU campus networking has made ipv6 progress...contacting ESNet to coordinate re: LHCONE

        Working on: OSG 3.5 upgrade, HTTP-TPC, NET2<->NESE networking 

         

      • 13:55
        SWT2 5m
        Speakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))

        UTA:

        Enabled new compute nodes (13 machines, 936 slots, ~12 KHS06)

        Enabled 6/11 new storage nodes, the other 5 are online but being used to sink data from retiring storage

        Otherwise everthing else is running well

        OU:

        Nothing to report

    • 14:00 → 14:05
      WBS 2.3.3 HPC Operations 5m
      Speakers: Doug Benjamin (Duke University (US)), lincoln bryant
    • 14:05 → 14:20
      WBS 2.3.4 Analysis Facilities
      Convener: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:05
        Analysis Facilities - BNL 5m
        Speaker: William Strecker-Kellogg (Brookhaven National Lab)
      • 14:10
        Analysis Facilities - SLAC 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:15
        Analysis Facilities - Chicago 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
    • 14:20 → 14:40
      WBS 2.3.5 Continuous Operations
      Convener: Ofer Rind
      • 14:20
        US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5m
        Speakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
      • 14:25
        Service Development & Deployment 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
    • 14:40 → 14:45
      AOB 5m