US ATLAS Computing Facility

→ US/Eastern
    • 12:00 → 12:10
      WBS 2.3 Facility Management News 10m
      Speakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn Mc Kee (University of Michigan (US))
    • 12:10 → 12:20
      OSG-LHC 10m
      Speakers: Brian Hua Lin (University of Wisconsin), Matyas Selmeci

      OSG 3.5 critical/bug fix support extended through April

      Release (this week)

      • vo-client-120 (updated DNs for FNAL, OSG, and GLOW)
      • osg-scitokens-mapfile-6 (IceCube and CMS token mappings)
    • 12:20 → 12:50
      Topical Reports
      Convener: Robert William Gardner Jr (University of Chicago (US))
      • 12:20
        TBD 30m
    • 12:50 → 12:55
      WBS 2.3.1 Tier1 Center 5m
      Speakers: Doug Benjamin (Brookhaven National Laboratory (US)), Eric Christian Lancon (Brookhaven National Laboratory (US))
      • a series of tests for the new library has been established. This list can be shared with NET2.
      • IPv4 issues on LHCONE seem to be gone, thanks Hiro the main driver of the investigations.

       

    • 12:55 → 13:15
      WBS 2.3.2 Tier2 Centers

      Updates on US Tier-2 centers

      Convener: Fred Luehring (Indiana University (US))
      • The last couple of weeks have been a good running period with fairly steady production.
        • Several incidents caused by central services but ADC has gotten pretty good at catching them and restoring the sites to production in a few minutes.
      • I am developing a spreadsheet to track Tier 2 site readiness for Run 3 data taking.
        • I have info for all sites but have only entered AGLT2 and MWT2.
      • OSG officially announced that the new EOL for OSG 3.5 is May 2.
        • This allows a less hurried update while providing a deadline well before the start of data taking.
      • Rob had an idea that we internally review the sites in preparation for the prescrubbing. This will get the information together that we need to present during the scrubbing process and also allow us to plan the FY22 purchases.
      • 12:55
        AGLT2 5m
        Speakers: Philippe Laurens (Michigan State University (US)), Dr Shawn Mc Kee (University of Michigan (US)), Prof. Wenjing Wu (University of Michigan)
        • UM vmware cluster lost an  MD3800i iscsi storage device(umvmstor03), so images of a few critical services, including afs server, cfengine, glpi, local squid servers were lost permanently, we had to rebuild all those services from scratch and restore the configuration from Amanda backup
        • MSU site Received 8x R6525 from September order (Rome 16C/32HT CPUs , 256G RAM).  
          Racked and running benchmarks (including boinc on/off and now working on hepscore).
          HS06 16.44/HT (no boinc) 
        • Did an update on one of the ATLAS gatekeepers (condor 9.0.8->9.0.9, condor-ce 5.1.2->5.1.3), all existing  jobs got lost on this gatekeeper (jobs in both condor and condor-ce sat in idle status for over an hour, had to remove all the jobs, and let new jobs come in.) For the other gatekeeper, we decided to drain it first before doing the update. 
        • For BOINC backfilling jobs, we changed the cgroup and cpu percentage used by BOINC jobs, it shows the impacts on grid jobs’ CPU efficiency drops by 6%, but the overall CPU utilization also dropped by 3%. 
        • Enabled WebDAV doors on all our existing dCache doors and changed CRIC to use them via the webdav.aglt2.org RR entry. Also configured the dCache doors to prefer IPV6 instead of IPV4 via http/webdav transfers.
      • 13:00
        MWT2 5m
        Speakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))

        UC

        • Phase two move this week. In the process of getting workers and storage back online
        • Upgraded condor on the workers to 9.x
        • Working on bringing up the new UC PerfSONAR nodes
        • Slowness on uct2-s28. Applied a couple of tunings that appeared to have helped. Applied the tunings to our other MD3460 nodes as well
        • Applying the configuration to prefer IPv6 to the dCache pools

        IU

        • Nothing to report

        UIUC

        • Upgrading condor on the workers to 9.x
      • 13:05
        NET2 5m
        Speaker: Prof. Saul Youssef
      • 13:10
        SWT2 5m
        Speakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))

        UTA_SWT2:

         We had a problem with the limited bandwidth into the cluster and the job mix since our fair-share allowed for some reconstruction jobs to  try to run.  We have set the Panda Queue to only run Evgen and Simul jobs and things have improved.

         

        SWT2_CPB:

        We closed the ticket related to transfers from RAL to SWT2_CPB.  Partly this was caused by the original configuration issues with the WebDAV service and the a desire by ddm-ops to seemingly move a large amount of data.  Hiro configured a link in FTS to provide enough priority to clear the backlog of transfers.

         

        OU:

        Site running well. We upgraded xrootd on se1 proxy gateway to 5.4.1 RC1, and so far, that's running well and hasn't hung or crashed.

         

    • 13:15 → 13:20
      WBS 2.3.3 HPC Operations 5m
      Speakers: Lincoln Bryant (University of Chicago (US)), Rui Wang (Argonne National Laboratory (US))

      TACC

      • Down to 15% of allocation remaining, have 6 weeks left.
      • Long queue times are slowing down our allocation burn.. a few mitigations in place
        • Harvester will now queue for 1 week. Previously it was configured to remove jobs after 48H which was killing us in getting through the queue. TACC prioritizes length-of-time-in-queue over other variables.
        • Reduced job size from 100 to 10 nodes. Smaller jobs should get through the queue faster.

      NERSC

      • Charging policy for NERSC is now as such:
        • Cori KNL "charge factor" is 0.2 vs 1.0 for Perlmutter in AY2022.
        • However, Perlmutter is still free until some date TBD, "likely in the second half of the year".
      • Given the above, we restarted jobs on Cori for the foreseeable future, but we should figure out how best to spend down the allocation once Perlmutter starts charging. 
      • Running jobs through NERSC_Perlmutter_Test queue currently, hoping to see a success soon. 
      • Rui and I will figure out optimal job sizing afterwards. Right now configured to use 128 (all) threads per job. Maybe do something similar to TACC and pack 16x 8-thread PanDA jobs per Slurm job. 
    • 13:20 → 13:35
      WBS 2.3.4 Analysis Facilities
      Convener: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 13:20
        Analysis Facilities - BNL 5m
        Speaker: Ofer Rind (Brookhaven National Laboratory)
        • Discussion of Discourse server last thursday - several updates/changes applied
        • BNL Jupyter federated ID technical issues have been resolved; finalizing processes and user environment/resources
      • 13:25
        Analysis Facilities - SLAC 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 13:30
        Analysis Facilities - Chicago 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

        Today is the 'grand opening' of the uc af.  Cecilia  Duran (Northern Illinois University) is presenting at today's IB meeting https://indico.cern.ch/event/1120534/. 

    • 13:35 → 13:55
      WBS 2.3.5 Continuous Operations
      Convener: Ofer Rind (Brookhaven National Laboratory)
      • OSG 3.5 EOL officially pushed back to May 1st.
      • dCache IPV4 preference was due to a jvm default setting (tracked down by Hiro, thanks!).  Simple fix implemented at BNL, though this is still not RFC compliant and dCache developers are working on a longer term fix.
      • FedOps: issues with AGLT2 SLATE squids (root partition usage) - looking into implementation of run status alerts; planning to migrate Drupal support pages to new US ATLAS web site
      • 13:35
        US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5m
        Speakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
      • 13:40
        Service Development & Deployment 5m
        Speakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

        We have a number of updates needed for squid devOps, and we need to setup training for a new squad of two, to relieve Lincoln.

        XCache

        • working fine. 
        • after a lot of discussions starting deployment of XCache in Birmingham to be used in VP context. depending how that will go we might see more sites in UK going cache only.

        VP 

        • working fine.
        • Some more parts of Rucio integration got into production

        ServiceX

        • changes in CI/CD
        • multipath support is ready for production next week. Will improve reliability.

        Analytics

        • new data streams from GeoModel
        • PerfSONAR changes working fine.
        • ES move will be completed before end of the week.
      • 13:45
        Kubernetes R&D at UTA 5m
        Speaker: Armen Vartapetian (University of Texas at Arlington (US))

        Today I got the hardware for initial k8s cluster. Will start working to setup k8s on it, and proceed with setting up k8s queue. We'll add more hardware once UTA_SWT2 hardware move is completed.

    • 13:55 → 14:05
      AOB 10m