US ATLAS Computing Facility

→ US/Eastern
Description

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  996 1094 4232

Meeting password: 125

Invite link:  https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09

 

 

    • 13:00 → 13:10
      WBS 2.3 Facility Management News 10m
      Speakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)

       

    • 13:10 → 13:20
      OSG-LHC 10m
      Speakers: Brian Lin (University of Wisconsin), Matyas Selmeci

      OSG 3.5.26 (not this week)

      • CVMFS 2.7.5
      • XRootD 4.12.5? RC1 is available in osg-testing, XRootD and OSG devs would appreciate any testing!

      Miscellaneous

    • 13:20 → 13:35
      Topical Reports
      Convener: Robert William Gardner Jr (University of Chicago (US))
      • 13:20
        No topical presentation today 15m
    • 13:35 → 13:40
      WBS 2.3.1 Tier1 Center 5m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
      • Smooth operation in general, except for a quick drop of running jobs last Wed, due to a PanDA bug.

      • core server migration of the HPSS tape system last Tuesday and Wed. went well, the frontend dCache was still up, impact on production is minimum

      • VP test ongoing, a couple of issues : permission issues on some dCache directories, under investigation; stage-in failures from BNLLake, to be excluded for now till further development from PanDA/Rucio

      • Users can't get files from BNLHPC_SCRATCHDISK -- datasets placed there by mistake in DDM, will be corrected.

    • 13:40 → 14:00
      WBS 2.3.2 Tier2 Centers

      Updates on US Tier-2 centers

      Convener: Fred Luehring (Indiana University (US))
      • Quiet week - no major outages on our side
        • Change to Panda put in during software week drained a large number of ATLAS sites world wide.
        • Still getting few GGUS tickets
      • Don't forget your reporting - please provide your reports a couple of days before the drop dead date so that the managers can see it before filing their reports.
      • Could everyone summarize their plans to spend the infrastructure/networking money.
        • I do understand we are meeting on this tomorrow but I would like to get a feeling of what the plans are.
      • 13:40
        AGLT2 5m
        Speakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)

        Incidents:

        1. One dcache pool went down due to controller memory corruption, rebooting and xfs_repair fixed the problem
        2. lost power in the entire T2 room on 2nd Oct due to overheating in the room and  overloading to the UPS. It took an entire day to resotre power and all services (switch and vmware services took long to recover). We fixed a failed power unit for the UPS, and retired 3 M610/M620 blade servers to save power usage in the Tier2 room. (ggus ticket 148900)
        3. lost 2 dcache pools (100TB space) due to aging hardware (3 years out of warranty)
        4. The major 100G switch connecting Tier2 and Tier3 crashed, rebooting fixed it
        5. ticket 148947: missing dump files for ATLASCALIBDISK. The scripts/cronjob/rpms generating and uploading the dump files were lost during our rebuilding of the dcache head nodes. It is restored
         
         
      • 13:45
        MWT2 5m
        Speakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))

        UC

        • New UC workers built and benchmarked. Should be online by the end of the week
        • HTTP-TPC configured as primary site protocol

        IU

        • Brocade debugging ongoing
        • Hardware issues with the IU SLATE node. Fred is working with Dell to resolve

        UIUC

        • Converging on a worker configuration for the current purchase cycle
      • 13:50
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))

         

        Xcache up and running on our NET2 slate node with help from Lincoln.  A corresponding ANALY_NET2_VP set up by Ilija and working fine. 

        Annual MGHPCC down day approaching on October 19.   All power will be out for NET2 and NESE. 

        Missing rucio file noticed in DAST list.  Apparently moved "to a WebDAV" enabled site?  

        Lots of planning happening for NESE tape. 

        Still a low level semi-issue with a bit more squid failovers than is comfortable (GGUS ticket).

        Smooth operation and full site (other than the PanDA dip that affected everyone) otherwise.

         

      • 13:55
        SWT2 5m
        Speakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))

        UTA:

        • Jobs running smoothly
        • Supporting ANALY-FRESNO-EKS-REMOTE with SWT2_CPB as data source/sink
        • Working on deploying recent storage purchases and retiring oldest storage

        OU:

        - Nothing to report, running well

        - HC failures at OU and BNL are caused by jobs not requesting correct RAM -- well, not requesting ANY RAM -- and therefore the pilot kills them because they are running over. Needs to be fixed on HC side.

         

    • 14:00 → 14:05
      WBS 2.3.3 HPC Operations 5m
      Speakers: Doug Benjamin (Duke University (US)), lincoln bryant

      TACC - Lincoln is on vacation this week. Prior to his vacation - Lincoln and Doug started to work splitting up TACC compute nodes to have 7 pilots per node (8 - cores per pilot).  

      NERSC - Cori is finally back but they still do not know the root cause of the FileSystem problems.  Running some small test jobs before  starting validation jobs.

      ALCF - started Test jobs on Theta. queue depth very deep jobs wait many days till they start. (almost a week) this could be an issue.

    • 14:05 → 14:20
      WBS 2.3.4 Analysis Facilities
      Convener: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:05
        Analysis Facilities - BNL 5m
        Speaker: William Strecker-Kellogg (Brookhaven National Lab)
      • 14:10
        Analysis Facilities - SLAC 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:15
        Analysis Facilities - Chicago 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

        All working fine. 

    • 14:20 → 14:40
      WBS 2.3.5 Continuous Operations
      Convener: Ofer Rind
      • 14:20
        US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5m
        Speakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
      • 14:25
        Service Development & Deployment 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
        • We are organizing a multi-site k8s federation platform for scalable analysis systems.  We held a first meeting yesterday with IRIS-HEP and CMS.  Looks like Admiralty can provide a lightweight federation layer. Plan is to connect up UC-River-dev, PRP, UW-Tiger, UNL-Tier2.  Will update here as things materialize.  URL: https://indico.cern.ch/event/965376/.  If you're interesting on participating, please send Ilija/Rob an email - all are welcome!

         

        • All US sites have working SLATE and XCACHE deployed. All sites have a VP queue, all accepting and finishing jobs. Caches will need ~ 1 month to warm up. BNL VP queue now gets payload files from AGLT_SCRATCHDISK. Getting it to upload results requires creating a completely new ATLAS site due to a limitation in Panda (all PQ at a site have to have the same write_lan:0 DDM endpoint).

         

        • Federated Ops
          • In context of SLATE, we have a set of security policy documents now that have been reviewed by OSG Security, US CMS Security, and US ATLAS Security (Shigeki):  https://slateci.io/docs/security-and-policies/index.html
          • These might be useful for local policy discussions at your facility

         

    • 14:40 → 14:45
      AOB 5m