US ATLAS Computing Facility (Possible Topical)

Wednesday 6 May 2026, 13:00 → 15:25 US/Eastern

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  993 2967 7148

Password: (previously shared to everyone in USATLAS)

Invite link:  https://umich.zoom.us/j/99329677148

 

 

13:00 → 13:05 WBS 2.3 Facility Management News

Speakers: Alexei Klimentov (Brookhaven National Laboratory (US)), Dr Shawn Mc Kee (University of Michigan (US))

FY26 funds for Tier-2s being distributed.   Waiting on end-of-CA funds

dCache workshop today and tomorrow (Eduardo and Shawn gave talks today)  https://indico.nikhef.nl/event/7562/ 

We should plan a focused procurement discussion sometime in the next ~2 weeks (prior to the due date for the plans)

 

13:05 → 13:10 OSG-LHC

Speakers: Brian Hua Lin (University of Wisconsin), Matyas Selmeci

• cvmfs-2.14.x in OSG upcoming, undergoing testing: includes a major rewrite to reduce capabilities held by the CVMFS client
• XRootD 6 work delayed due to CopyFail
• HTCondor 25.10.1 and 25.0.10 expected next week
  • https://htcondor.readthedocs.io/en/main/version-history/feature-versions-25-x.html#version-25-10-1
  • https://htcondor.readthedocs.io/en/main/version-history/lts-versions-25-0.html#version-25-0-10

13:10 → 13:30 WBS 2.3.1: Tier1 Center

Convener: Alexei Klimentov (Brookhaven National Laboratory (US))

13:10 Tier-1 Infrastructure

Speaker: Jason Smith

13:15 Compute Farm

Speaker: Thomas Eric Smith (Brookhaven National Laboratory (US))

All systems were remediated and rebooted late last week to address linux vulnerability

Completion of migration of Tier 1 to new condor configuration/ puppet code, which is working well

Small modification to condor Fifemon to fix missing data in our internal grafana monitoring

13:20 Storage

Speakers: Carlos Fernando Gamboa (Brookhaven National Laboratory (US)), Carlos Fernando Gamboa (Department of Physics-Brookhaven National Laboratory (BNL)-Unkno)

13:25 Tier1 Operations and Monitoring

Speaker: Ofer Rind (Brookhaven National Laboratory)

• We are still dealing with high-load induced on our shared storage (since April 8th) that resulted in large spikes of job failures since last Friday (subsequent to the copy-fail mitigation reboots).  This appears to be the same issue discussed and introduced in this GGUS ticket created by Andrey.  Still trying to get some clear guidance about how best to address it, although pilot fixes have already been applied in testing.  There is a Jira ticket following this issue:  https://its.cern.ch/jira/browse/ATLASPANDA-1736. 

13:30 → 13:40 WBS 2.3.2 Tier2 Centers

Updates on US Tier-2 centers

Conveners: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

• Good running over the past two weeks
  • AGLT2 took a downtime to do system upgrades on 4/30
  • MWT2 had an unintended downtime while mitigating copy fail CVE.
• No news yet about end of CA equipment funding.
• dCache updates to version 11.2.4 which has Fireflies/SciTags enabled and working correctly:
  • AGLT2 went to the most recent dCache version (11.2.4) yesterday.
  • MWT2 will take a downtime to update to 11.2.4 next Monday.
• Shawn reminded me that we need to have a meeting to discuss procurement.
  • Memory prices have been stable (and high!) since February.
• Benchmarking indicates that buying from Dell, that 5th generation EPYC CPUs get better numbers .
  • 9355   128HT Cores 300W/CPU
  • 9374F 128HT Cores 400W/CPU

13:40 → 13:50 WBS 2.3.3 Heterogenous Integration and Operations

HIOPS

Convener: Rui Wang (Argonne National Laboratory (US))

13:40 HPC Operations

Speaker: Rui Wang (Argonne National Laboratory (US))

Perlmutter:

• Production job running smoothly
  • (Doug) Investigating the empty SLURM jobs: short-lived but affecting the SLURM job priority as only two jobs are allowed per user
• ~50% GPU allocation used

 

13:45 Integration of Complex Workflows on Heterogeneous Resources

Speaker: Doug Benjamin (Brookhaven National Laboratory (US))

13:50 → 14:10 WBS 2.3.4 Analysis Facilities

Convener: Wei Yang (SLAC National Accelerator Laboratory (US))

13:50 Analysis Facilities - BNL

Speaker: Qiulan Huang (Brookhaven National Laboratory (US))

• Testing the new jupyterhub front-end is undergoing.  (https://jupyter-fedid.sdcc.bnl.gov)

  • uses Federated ID, requires ORCID

  • CPU notebooks are working well, while GPU instances are spawning more slowly.

13:55 Analysis Facilities - SLAC

Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))

14:00 Analysis Facilities - Chicago

Speaker: Fengping Hu (University of Chicago (US))

• Copy Fail (Apr 30)
  Mitigated by blacklisting the algif_aead initcall via grubby.
• Coffea-Casa
  Deployed new image: coffea 2026.4.0.
  Removed older image (2025.9.0) from the menu; users are encouraged to migrate to the latest version.
• AI Agent in Operations
  Piloting a CLAW-based agent to monitor HTCondor job queue anomalies and provide corrective suggestions.
  Notifications are sent via email to users, with a human-in-the-loop for approval.

14:10 → 14:30 WBS 2.3.5 Continuous Operations

Conveners: Ivan Glushkov (Brookhaven National Laboratory (US)), Ofer Rind (Brookhaven National Laboratory)

14:10 ADC Operations, US Cloud Operations: Site Issues, Tickets & ADC Ops News

Speaker: Kaushik De (University of Texas at Arlington (US))

• BNL
  • Errors due to multiple files left by the ALRB in the home directory. To change the pilot to write only in the WORKDIR of a job. SWT2 opened ticket about this one month ago.
  • HC HS23 jobs are failing. The responsible is looking into it (she was on leave the last week)
• MWT2
  • Basepath correction postponed for Monday (GGUS:1002243)
• SLAC
  • Failing transfers (GGUS:1002576)
• OU
  • Dual stack finally deployed on grid1.oscer.ou.edu!  (GGUS: 683424)
• Misc
  • NERSC_LOCALGROUPDISK - reconfigured (to use davs insted of https, ATLDDMOPS-5830)
  • NERSC_DATADISK - not used, but agreed by DDM Ops to leave it for potential testing of Rucio with Globus
  • NEVIS localgroupdisk - decommissioned (ATLDDMOPS-5828)

14:15 Services DevOps

Speaker: Ilija Vukotic (University of Chicago (US))

• XCaches
  • image needs a rebuild.  osg rpm doesn't include for example voms2-atlas-auth.cern.ch.lsc
  • need to send our config to Doug
  • otherwise, instances run fine
• VP
  • works fine
• Varnishes and Frontiers
  • all work fine
• AI
  • we started a big cleanup and refactoring of our repositories
  • starting to move agents to nemoclaw sandboxes 
• Analytics
  • restart due to updates broke a few logstash instances. Fixed for now, but will need a bit of development to prevent such issues in the future.

14:20 Facility R&D

Speaker: Robert William Gardner Jr (University of Chicago (US))

14:25 Cybersecurity plan(s)

Speakers: Robert William Gardner Jr (University of Chicago (US)), Shigeki Misawa (Brookhaven National Laboratory (US))

14:30 → 14:40 AOB