US ATLAS Computing Facility (Possible Topical)

Wednesday 9 Sept 2026, 13:00 → 15:25 US/Eastern

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  993 2967 7148

Meeting password: 452400

Invite link:  https://umich.zoom.us/j/99329677148

 

 

13:00 → 13:05 WBS 2.3 Facility Management News

Speakers: Alexei Klimentov (Brookhaven National Laboratory (US)), Dr Shawn Mc Kee (University of Michigan (US))

• Alexei is traveling today
• Still waiting for End-of-CA funds and for scrubbing final results
  • Hoping the NSF will allow a no-cost extension to mitigate the Dell supply chain lead time issue
• Need to plan out mini-challenges leading up to DC27
  • Relevant meetings for discussion include LHCONE/LHCOPN and WLCG/HSF meetings 
  • Need to identify a good time for a USATLAS discussion, perhaps tomorrow afternoon or next Thursday or Friday?
• Please consider attending HEPiX in Lincoln Nebraska.  We are trying to create two lunch meetings to have discussions with USCMS on areas of joint interest.
• Next week is the CLARIPHY AI collaboration meeting in Madison:  https://indico.global/event/18639/

 

Zoom AI Notes

Quick recap

This meeting focused on updates and discussions across various ATLAS facilities and operations. Frederick reported delays in receiving paperwork from Chicago for Midwest Tier 2 funding and discussed ongoing procurement challenges for storage equipment, noting delays in receiving configurations from vendors. The team discussed upcoming facility meetings including LHCOPN in Naples and WLCG HSF in Bologna, with plans to schedule a USATLAS discussion meeting for facility planning and challenges leading up to DC27. Ofer promoted the upcoming HEPICS conference in Lincoln, Nebraska, highlighting discussions with USCMS and a special session on Kubernetes platforms. Brian reported that OSG 26 development was progressing with a target end-of-month release, while Thomas updated on the rolling upgrade of the ATLAS tier one farm from Alma 9.7 to 9.8. Carlos provided updates on DCache installation upgrades and successful testing of the staging workflow, while Ofer mentioned upcoming electrical power maintenance and the draining of opportunistic queues for S-Phoenix. Ivan reported three significant issues in the 2-3-5 system including token support problems, an IDTF library issue affecting authentication, and a critical FTS4 production error that caused data corruption affecting 376K files. Ilija discussed recent fixes to various services and mentioned upcoming collaboration on AI agents for DC27 with CMS colleagues.

Next steps

Frederick

• Rent a car from the motor pool and attend the meeting on Tuesday the 22nd.
• Send procurement questions to Dell regarding storage configurations and follow up on delays.
• Arrange a meeting with Dell (and a few attendees) to discuss cost-effective hardware options and parts availability.
• Add Jordan to the WBS 2,3 mailing list and Google group, and notify him.

Ilija

• Renew all the certificates used by the excaches and send them to the relevant people.
• Change the code to run Data Challenge for monitoring and cooperate with CMS on AI agents for DC 27.

Ofer

• Coordinate the USCMS meeting at HEPICS with the OTF follow-up on facilities.

Shawn

• Send out a Doodle poll to the WBS 2,3 USATLAS mailing list to schedule a meeting to discuss facilities, USATLAS plan, and the road to DC 27.
• Attend the AI collaboration meeting in Madison next week.
• Share the GPU survey with more ATLAS users beyond the Google group.
• Emphasize to Dell at the upcoming meeting the critical need for continued support of OpenManage/OMSA.

Summary

Funding and Meeting Updates

Frederick informed Shawn that he is waiting for two increments of paperwork from Chicago regarding his Midwest Tier 2 funding, which ended on January 31st, though he has sufficient equipment funding to continue operations. The team discussed concerns about Dell's OpenManage server administrator tool reaching end-of-life in September 2027, noting this contradicts previous statements from Dell and investigating alternative solutions like Dell ONE and OME. Shawn announced upcoming important meetings including the LHCOPN meeting in Naples and WLCG HSF meeting in Bologna, and suggested scheduling a USATLAS discussion for either the current day or next Thursday or Friday, with Friday lunchtime being a preferred time for many team members.

Facility Survey and AI Collaboration

Ofer announced plans to coordinate with the USCMS meeting regarding an upcoming facility survey and discussed a special session on Kubernetes platforms. Shawn mentioned an AI collaboration meeting in Madison next week and noted that Jordan is planning presentations there. Frederick suggested getting Jordan more involved in future meetings due to his expertise in AF-related matters.

OSG 26 Development Progress Update

The team discussed OSG 26 development progress, with Brian reporting they are working toward an end-of-month release and implementing earlier testing through the Condor dev team. Thomas provided an update on the Atlas tier one farm upgrade, which is approximately 80-90% complete with an OS upgrade from Alma 9.7 to 9.8. The team noted upcoming downtime at BNL over the weekend affecting the shared pool, including ATLAS Tier 3 and analysis facilities, with potential impacts on Monday.

Detach Installation Upgrade Plans

Gamboa announced that the Detach installation will be upgraded to version 11.2.7 on Monday, which includes support for G plasma and bonds configuration for SHA1 certificates. The HPCS team successfully upgraded the batch component of the staging workflow, with ATLAS generating 30% of the 62,000 staging requests that successfully completed. The integration instance validation of the latest mining release on DCache showed no issues, and the infrastructure team is working on commissioning the newly arrived JBODs for the pool servers.

System Maintenance and Updates

Ofer announced electrical power supply maintenance scheduled for Saturday, which will run the facility on generator power with extensive testing completed over the past two weeks. Carlos mentioned the DCACHE update downtime and the opportunistic queues on the shared pool stopping new pilots on Sunday morning for S-Phoenix drainage. Frederick reported good overall system performance over the past weeks, with the notable exception of Midwest Tier 2 experiencing series problems and a minor incident at ADC causing a temporary drop in slots across sites. Frederick also updated on ongoing procurement work for storage, mentioning delays in receiving configurations and plans to follow up with vendors after receiving feedback from Philippe, Robert Hancock, and Eduardo.

Head Node Server Implementation Discussion

Frederick and Shawn discussed the implementation of head node servers for Michigan, with Shawn confirming they were already in place. They discussed waiting for Dell to provide insights on parts availability and cost-effective alternatives. Frederick proposed scheduling a meeting with Dell to discuss sweet spots and cost-effectiveness questions. Frederick also mentioned that after reviewing obligations, Chris and John discovered they had approximately $2 million less than expected, with a total of about $5.5 million available to be split between four Tier 2 federations and the AF.

HPC Resources and System Updates

Rui reported that HPCs have only 42K CPU node hours remaining, lasting about one week at current rates, and will switch to a round queue unless Wahid and Jack provide alternative support approaches from NERSC. The GPU survey has been submitted to Google Group and USATLAS mailing list, with plans to share it more widely since not all ATLAS users are on the Google group. Qiulan updated on BL facilities work with Luis on Jupyter Puppet code debugging and testing, noting AF SharePool node upgrades scheduled for Monday. Ivan reported three key issues: token support problems with PITR tests, an IDTF Java library problem that was quickly fixed by releasing a previous version, and an FTS4 production issue that was resolved by switching back to FTS3, resulting in 376K affected files including some on tapes, with approximately 60K files still lost and under investigation.

Production Issue and System Updates

Ivan explained that a production issue occurred due to a version being accidentally deployed that was not supposed to go live, compounded by the IGTF incident making it difficult to distinguish the problem. He noted that the issue only affected certificate users and was further complicated by different storage systems' responses, with some refusing to overwrite files while others did not. Ilija reported that several issues with tax cash have been resolved, and mentioned that search certificates need renewal, though not immediately. He also confirmed that Elasticsearch, printer, and other services under his responsibility are functioning properly, including the virtual placement system which now has redundant clusters.

CMS AI Agents Meeting Planning

Ilija confirmed that he met with Shawn and Marianne about CMS AI agents for DC 27 and will modify code to run the Data Challenge for monitoring. Shawn announced he would send a Doodle poll to schedule a meeting about the USATLAS facilities plan for many challenges and DC 27, with options for the 10th, 17th, or 18th. The meeting will discuss capability and capacity tests, purchasing decisions based on previous challenges, and site-specific infrastructure needs. Frederick suggested including Rafael in the meeting due to his interest in these topics.

13:05 → 13:10 OSG-LHC

Speakers: Brian Hua Lin (University of Wisconsin), Matyas Selmeci

Heads down on OSG 26!

13:10 → 13:30 WBS 2.3.1: Tier1 Center

Convener: Alexei Klimentov (Brookhaven National Laboratory (US))

13:10 Tier-1 Infrastructure

Speaker: Jason Smith

13:15 Compute Farm

Speaker: Thomas Eric Smith (Brookhaven National Laboratory (US))

Rolling upgrade of atlas T1 farm is nearing the end (OS upgrade - Alma 9.7 to 9.8)

Upgrades to shared pool (atlas T3/AF) forthcoming, Monday 14 Sep. This will include OS upgrades and reconfiguration of HTCondor to modern standard

13:20 Storage

Speakers: Carlos Fernando Gamboa (Brookhaven National Laboratory (US)), Carlos Fernando Gamboa (Department of Physics-Brookhaven National Laboratory (BNL)-Unkno)

dCache Upgrade

• The production dCache upgrade to version 11.2.7 is scheduled for September 14, 2026, from 9:00 AM to 5:00 PM EDT.

 

HPSS and Batch Services

• HPSS maintenance was completed on September 8, 2026, between 9:30 AM and 4:00 PM.

• The batch service was migrated to a NetApp NFSv4 service.

• The new batch system was upgraded.

• No issues have been observed so far.

• More than 62,000 staging requests were successfully delivered, with approximately 30% of the load generated by ATLAS.

 

Integration Instance

• dCache 11.2.7 was deployed on the integration instance.

      The gplazma2-voms configuration was updated to enforce OpenSSL SHA-1 mode for certificate hash calculations.

 

Hardware Procurement

• The new JBODs arrived at the laboratory on September 4, 2026.

• Hardware commissioning is underway.

 

13:25 Tier1 Operations and Monitoring

Speaker: Ofer Rind (Brookhaven National Laboratory)

• Planning for a non-disruptive electrical power supply maintenance on Saturday
  • Will run facility on generator power after extensive testing over the past two weeks
  • Experts on call in case any issues arise 
• dCache update downtime scheduled (via Topology) for next Monday 9am-5pm
  • Opportunistic HTCondor queues on shared pool will stop accepting new pilots on Sunday 9am
• HPSS batch server was upgraded (RH8 VM, NetApp NFS shared dir) on Tuesday with no disruption to operations

13:30 → 13:40 WBS 2.3.2 Tier2 Centers

Updates on US Tier-2 centers

Conveners: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

• Good running over the past couple of weeks.
  • MWT2 had network problems at the UC site
• Still banging away at the procurement work for the storage.
  • Will send further questions Dell later today.
• End of CA paperwork can be sent out but last minute negotiations continue.
  • The amount of end of CA funding is less than we expected.

13:40 → 13:50 WBS 2.3.3 Heterogenous Integration and Operations

HIOPS

Convener: Rui Wang (Argonne National Laboratory (US))

Perlmutter: 42K CPU node hours left (~1 week run), will switch to overrun queue if no news from Wahid and Jack

• GPU survey to the Perlmutter GPU Google group and the USATLAS mailing list
  • Could be good to share with ATLAS users
• (Doug) updated the pilot

13:40 HPC Operations

Speaker: Rui Wang (Argonne National Laboratory (US))

13:45 Integration of Complex Workflows on Heterogeneous Resources

Speaker: Doug Benjamin (Brookhaven National Laboratory (US))

13:50 → 14:10 WBS 2.3.4 Analysis Facilities

Convener: Wei Yang (SLAC National Accelerator Laboratory (US))

13:50 Analysis Facilities - BNL

Speaker: Qiulan Huang (Brookhaven National Laboratory (US))

• Work on debugging and testing the Jupyter Puppet code
• Plan the downtime for the AF computing nodes upgrade on next Monday

13:55 Analysis Facilities - SLAC

Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))

14:00 Analysis Facilities - Chicago

Speaker: Fengping Hu (University of Chicago (US))

14:10 → 14:30 WBS 2.3.5 Continuous Operations

Conveners: Ivan Glushkov (Brookhaven National Laboratory (US)), Ofer Rind (Brookhaven National Laboratory)

• Status of storage token support at US sites?
  • Petr tests look ok, but Rucio attributes indicate otherwise.  Can this be reconciled?
    • Petr's test are not necessarily comprehensive. Enabling tokens was tried for SWT2_CPB_SCRATCHDISK in the past 
    • Yes, DDM is contacting sites when switch to token is planned
• Summary of 9/1 IGTF 1.145 release incident (link)
  • dCache patched for now, but canl-java will still need a more lasting update
    •  "Campaigns will need to be organized for sites to update or upgrade to releases with those fixes"
• FTS4 incident on Thursday lead to files being zeroed out (including on tape).
  • 376k files involved
  • only 6k are disk replicas actually lost

14:10 ADC Operations, US Cloud Operations: Site Issues, Tickets & ADC Ops News

Speaker: Kaushik De (University of Texas at Arlington (US))

14:15 Services DevOps

Speaker: Ilija Vukotic (University of Chicago (US))

14:20 Facility R&D

Speaker: Robert William Gardner Jr (University of Chicago (US))

14:25 Cybersecurity plan(s)

Speakers: Robert William Gardner Jr (University of Chicago (US)), Shigeki Misawa (Brookhaven National Laboratory (US))

14:30 → 14:40 AOB