US ATLAS Computing Facility

→ US/Eastern
Description

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  996 1094 4232

Meeting password: 125

Invite link:  https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09

 

 

    • 13:00 → 13:10
      WBS 2.3 Facility Management News 10m
      Speakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
    • 13:10 → 13:20
      OSG-LHC 10m
      Speakers: Brian Lin (University of Wisconsin), Matyas Selmeci

      OSG 3.5.25 (tomorrow)

      • CVMFS 2.7.5
      • Frontier Squid in EL8

      Miscellaneous

    • 13:20 → 13:35
      Topical Reports
      Convener: Robert William Gardner Jr (University of Chicago (US))
    • 13:35 → 13:40
      WBS 2.3.1 Tier1 Center 5m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
      • Smooth operation in general
      • MAS R&D -- Evaluation and deployment of the new dCache feature -- make extra tape replica of files-on-disk, transparently to users -- DATADISK stay as hot storage (2 disk copies). BNLLAKE_DISK as intermediate layer (1 disk copy + 1 tape copy)
    • 13:40 → 14:00
      WBS 2.3.2 Tier2 Centers

      Updates on US Tier-2 centers

      Convener: Fred Luehring (Indiana University (US))
      • Good running until recently.
        • 3 GGUS tickets open
        • Storage problems today at AGLT2, MWT2, & NET2 (Minor CPB issues solved a few days ago)
      • Putting HTTP-TPC as the first protocol at MWT2 on hold
      • SLATE server at all sites milestone nearly done
      • Infrastructure/network funding? Soon I guess....
      • ipv6 status?
      • 13:40
        AGLT2 5m
        Speakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)

        Tickets:
        148265: dcache storage went offline, The problem was traced to two of our zookeeper instances having filled their var partitions thus losing quorum. We added compression to their logrotate
        and that should prevent the problem from re-occurring.
        14829: dCache Transfer failing, this is caused by one dcache pool node having timeout transfer), we replaced the link between 2 main switches, which significantly improved the transfer rate to this pool node.
        148666: dCache low transfer efficiency. One of the pool nodes lost one of the system disks, it crashed at trying to reboot itself overnight.

        Hardware:
        All UM work nodes and dcache pool nodes from 2020 new purchase are in production. MSU nodes are still in process.

        Software update:

        www.aglt2.org and wiki rebuilt and re-allowed through campus firewall
        squid-frontier to 4.13-1.1
        check-mk 1.6p17
        ELK cluster 7.9.1, added 4 powerful nodes to the cluster. (40Gpbs link, 1TB RAM, 2*3TB Nvme cards)

        7 AFS servers all migrated from SL6 to CC7. 

         

         

         
         
      • 13:45
        MWT2 5m
        Speakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))

        Multiple dcache issues trying to update to 5.2.28 so we downgraded and are sitting on 5.2.25 

        One of our storage machines had a controller crash over the weekend. We brought back up and there haven't been issues since.

        Missing files GGUS ticket: 148601 we are investigating to try and pinpoint the cause. Brought this up with the dcache team.

        Errors have dropped back down and we're in working condition again.

        UC:

        • Waiting on several new compute nodes to be delivered and getting our last bit of misc. hardware necessary to start getting our new compute online.
        • Storage issues. Working on pinpointing and fixing.

        IU:

        • Currently building new compute. Over half is built and in production.
        • Getting compute on IPv6 in waves

        UIUC:

        • Working on getting down compute back up and running.
      • 13:50
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))

         

        Added worker nodes.

        Joined SLATE federation.  Thanks for Lincoln for a consultation.

        Added ~3PB of NESE Ceph storage.

        ~1000 panda jobs had stage-out timeouts yesterday.   Still investigating.   

        Smooth operations otherwise. 

      • 13:55
        SWT2 5m
        Speakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))

        UTA:

        • Deployed 1.5 PB of storage, final testing on another 1.5 PB, more storage to come
          • Will retire very oldest storage (MD1x000 based systems): ~600TB
        • Awaiting internal UTA delivery of latest compute nodes, maybe today
        • Deployed SRR reporting script for SWT2_CPB
          • Waiting to hear if the JSON is acceptable.
          • Will deploy to UTA_SWT2 and OU

        OU:

        - Nothing to report, all running well.

        - Getting great opportunistic throughput right now, up to 1000 additional slots.

         

    • 14:00 → 14:05
      WBS 2.3.3 HPC Operations 5m
      Speakers: Doug Benjamin (Duke University (US)), lincoln bryant
    • 14:05 → 14:20
      WBS 2.3.4 Analysis Facilities
      Convener: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:05
        Analysis Facilities - BNL 5m
        Speaker: William Strecker-Kellogg (Brookhaven National Lab)
      • 14:10
        Analysis Facilities - SLAC 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:15
        Analysis Facilities - Chicago 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
    • 14:20 → 14:40
      WBS 2.3.5 Continuous Operations
      Convener: Ofer Rind
      • SLATE deployed at BU last week.  Status of Xcache?
      • Patrick and Lincoln working on SLATE install at CPB
      • HTTP-TPC changeover at MWT2 still being held up by lingering problems after dCache upgrade
      • Some discussion at yesterday's ADC weekly re: best use of new upcoming NESE tape storage and how the storage element will be accessed (need HTTP-TPC solution)
      • Mark, with Fred and Ofer, putting together a facility-wide service profile spreadsheet (sw versions, capabilities)
    • 14:40 → 14:45
      AOB 5m