US ATLAS Computing Facility (Possible Topical)

Wednesday 23 Sept 2026, 13:00 → 15:25 US/Eastern

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  993 2967 7148

Meeting password: 452400

Invite link:  https://umich.zoom.us/j/99329677148

 

 

13:00 → 13:05 WBS 2.3 Facility Management News

Speakers: Alexei Klimentov (Brookhaven National Laboratory (US)), Dr Shawn Mc Kee (University of Michigan (US))

Alexei and Ofer had a critical meeting conflict and can't attend.  A few important items for today:

• End-of-CA funds and procurement (see Tier-2 section for details)
  • Met with Dell yesterday at Michigan
  • All Tier-2s need to get their 1030s, Justifications and SOWs in to Stony Brook ASAP
  • Fred is working on getting initial quotes representing all intended Dell purchases
• HEPiX is coming up AND today is the end of the Early Bird registration...have your registered?
  • Ofer would like a quick headcount of those intending to register BUT have not yet done so
• Today was an ATLAS meeting about DC27:  https://indico.cern.ch/event/1731149/
  • ATLAS has a good estimate of its needed HL-LHC rate (see Alessandra's slides)
  • Lots to discuss at upcoming DOMA (Sep 30) and WLCG/HSF workshop in beginning of November
• CHEP paper deadline this Friday!!

 

ZOOM AI NOTES

 

Quick recap

This meeting focused on updates and coordination across USATLAS Tier 1 and Tier 2 centers regarding equipment procurement, storage operations, and upcoming events. Shawn announced that Dell meetings were held to discuss critical equipment purchases, with all Tier 2 centers needing to submit required paperwork including justifications and statements of work to Stony Brook as soon as possible to enable purchase order issuance. The team discussed HEPICS registration deadlines, with Gamboa confirming he would register, and noted that abstract submissions could continue until October 6th. Frederick provided detailed updates on Dell negotiations, explaining that pricing would be based on total spend across USATLAS rather than individual sites, and that Dell was willing to provide both 3-month costed extension and no-cost extension options. The meeting covered various operational updates including storage upgrades, tape library maintenance, CPU allocations at NERSC, and recent site issues with blacklisting and drops. Key technical discussions included network requirements for AI/ML inference services and the need for CyTags marking 80% of traffic during DC27 operations.

Next steps

Douglas

• Reach out to Eric Fang to inquire about additional CPU/GPU hours for Perlmutter.
• Shape and manage user jobs on Perlmutter to ensure throughput in the overrun queue.

Frederick

• Collect and consolidate the full purchase plan (servers, switches, etc.) for Dell and request final pricing quotes.
• Get quotes from Dell for the two specific switches (management and data) for Shawn's site.
• Request a refresh of the existing quotes from Dell to confirm feasibility and pricing.

Ivan

• Look into implementing rules for what users are allowed to stage from tape.

Rui

• Write up instructions for parameter users submitting local jobs under the ADC documentation page.

Shawn

• Submit CHEP 2026 papers by the deadline (Friday).

Collaboration

• Tier 2 Sites: Submit 1030s, justifications, and statements of work to Stony Brook as soon as possible.
• Shawn & Frederick: Coordinate with Dell to finalize pricing and enable sites to place purchase orders within 14 days.
• Kaushik/Ivan: Include NET2 in the analysis of the effect of additional Tier 2 tape sites on throughput.
• Shawn & Frederick: Consider scheduling a meeting with Rob to ensure paperwork and procurement are progressing.

Summary

CA Fund and Procurement Updates

Shawn opened the meeting and noted that Alexei and Ofer were unable to attend due to a scheduling conflict. He announced that Fred would discuss end of CA fund information and associated procurements, highlighting the need for Tier 2s to submit their justifications and statements of work to Stony Brook promptly to enable purchase orders. Shawn also reminded attendees about the upcoming HEPICS conference, mentioning the end of early bird registration and a planned USATLAS-US-CMS session, though the specific timing was still tentative.

Event Registration and Rate Calculations

The team discussed upcoming registration deadlines for an event, with abstracts due October 6th and early bird registration being considered for extension. Shawn reported on an ATLAS meeting about DC 27 where new rate calculations were presented, and mentioned upcoming DOMA and WLCG HSF workshops where DC 27 will be discussed, including potential postponement considerations. Douglas raised a question about AI/ML implications for network usage, particularly regarding inference operations, which the team acknowledged as an unknown factor that would need further estimation to determine its impact on rates.

Triton Server and Release Updates

Rui discussed Cheyenne's Triton server study with Elcf and suggested collaboration on performance testing. Shawn reminded the team about the CHEP 2026 paper deadline on Friday. Brian announced that the OSG 26 release is delayed and will now ship in mid-October. Gamboa provided updates on storage, including a successful dcache upgrade and ongoing work on commissioning new servers, as well as addressing transfer issues and upcoming tape library maintenance.

HPSS Downtime and File Servers

The team discussed HPSS downtime and file server capabilities, with Carlos confirming that not all HPSS would be affected. Doug inquired about new file servers' ability to send fireflies directly, and Carlos explained the policy for new hardware procurement in the storage element. Shawn mentioned the goal of having 80% of traffic during DC27 marked by CyTags, with BNL potentially having a lower fraction and other sites closer to 100%. Frederick indicated he had a long update about tier two centers and requested that people not interrupt him while presenting the details.

Dell Pricing and Operations Discussion

Frederick reported that operations have been running smoothly for the past couple of weeks despite some production losses at Midwest Tier 2 due to disruptions. He met with Dell's higher-level sales managers to discuss pricing, with Dell still willing to offer price reductions if they can see the full purchase plan including server and switch quantities. Frederick and Shawn agreed to collect the list of items for quoting, with Dell requesting a 14-day PO commitment while working to secure longer-term pricing due to weekly supplier price increases.

Storage and Infrastructure Updates

Shawn and Frederick discussed storage options, with Shawn explaining that they prefer simpler storage solutions without extra features that would need to be ignored. Frederick mentioned receiving information from Brian at Geodecki about treating USATLAS as a single entity for consistent pricing and better discounts based on total spend. Frederick also shared his positive experience purchasing PDUs from Dell at a competitive price. Rui provided updates on Edge PCs, noting that CPU allocation for Perlmutter has been exhausted and they are requesting top-up from DOE for both CPU and GPU, while also working on Slurm worker settings and documentation for parameter users.

NERSC Resource Allocation Updates

Douglas reported that he reached out to Eric Fang about getting additional hours, but the NERSC reserve for OHEP is exhausted, so they may need to shut off users' CPU access and shape jobs by timeline. The team discussed analysis facilities reports from BNL in Chicago, with Thomas indicating he is still working on it and plans to conduct testing on JHub05.

System Maintenance and Updates Review

The team discussed recent maintenance activities and system updates. Fengping reported that quarterly maintenance was completed successfully and MTN DNS tunings improved cluster stability with fewer alerts. Kaushik addressed issues with DC27 blacklisting and site drops, advising teams to check if problems affect all sites when experiencing similar incidents. The team also noted that recent storms and power outages in the central U.S. impacted operations but were managed with minimal effect.

Job Update and Testing Plans

Ivan discussed a recent issue where jobs were not being updated, which led to HammerCloud blacklisting sites. He took responsibility for not sending any mail notifications during the incident on Saturday, though the problem was resolved within several hours by other team members. Ivan outlined plans to include NET2 in future testing to evaluate the impact of additional Tier 2 tape sites on performance, and he mentioned exploring rules for user tape staging permissions.

Equipment Purchase Extension Planning

Shawn explained that they will receive both a 3-month costed extension and a no-cost extension for equipment purchases, though the no-cost extension lacks official paperwork. Aidan inquired about this extension due to planned network equipment purchases from Arista, and Shawn confirmed that if both extensions proceed, shipping could be completed by April 2028. After the meeting, Frederick expressed concerns about Rob's progress and suggested having a follow-up meeting with him, to which Shawn agreed but noted that Rob should first have time to process the recent flurry of paperwork.

 

13:05 → 13:10 OSG-LHC

Speakers: Brian Hua Lin (University of Wisconsin), Matyas Selmeci

OSG 26 delayed to ~mid October

13:10 → 13:30 WBS 2.3.1: Tier1 Center

Convener: Alexei Klimentov (Brookhaven National Laboratory (US))

13:10 Tier-1 Infrastructure

Speaker: Jason Smith

13:15 Compute Farm

Speaker: Thomas Eric Smith (Brookhaven National Laboratory (US))

Tier 1 has been solid - only issue was the mass draining event on 19 SEP, but this was not a site issue

Tier 3/AF - I've been working on the HTCondor side of the jupyter pool config to bring it up to the modern standard of the rest of the shared pool

13:20 Storage

Speakers: Carlos Fernando Gamboa (Department of Physics-Brookhaven National Laboratory (BNL)-Unkno), Carlos Fernando Gamboa (Brookhaven National Laboratory (US))

dCache Upgrade

• The major upgrade of the BNL US ATLAS dCache service from version 9.2.x to 11.2.7 was successfully completed on September 14.

New Pool Servers and JBODs

• The new pool servers and associated JBODs are being commissioned for production.

• Integration into production workflows is currently underway.

• The new configuration provides isolation of pool metadata from the primary pool storage.

Operations

• Addressed GGUS ticket 1003953, concerning transfer issues when accessing some staged files.

HPSS Maintenance

• An intervention on the ACSLS tape-library server is scheduled for September 24, 2026 9:00 AM–1:00 PM.

• The maintenance will affect ATLAS staging requests for data written more than five years ago.

• Affected requests will remain queued and resume automatically after the maintenance is completed.

 

13:25 Tier1 Operations and Monitoring

Speaker: Ofer Rind (Brookhaven National Laboratory)

13:30 → 13:40 WBS 2.3.2 Tier2 Centers

Updates on US Tier-2 centers

Conveners: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

• Good running over the past two weeks with some production reductions:
  • MWT2 lost some production doing a rolling update of condor
  • Database problems at CERN cause some production loss over last weekend.
• Please see the top of the meeting notes for important information about the needed paperwork.
• Shawn, Wenjing, Philippe and Fred met for an hour with 2 higher level Dell sales managers yesterday, Tuesday, 9/22/2026. The following issues were discussed:
  
  

1. Dell attendees: Brian Teodecki (Manager of 8 university sales teams) and Mitch Breneman (Financial manager). Fran Bogle US VP for higher ed got caught by the east coast air traffic problems and could not attend.
2. Dell is still willing to reduce the price further but they need to see our full purchase plan (how many servers, how many switches). For the storage we can get a still lower price if we order a lot of one configuration. They were clear that the final pricing will depend on the total buy size e.g. $4 million but if we tell them an amount  then we need to stick with it and individual universities can't try to wring additional price concessions from Dell.
3. We (Shawn and Fred) agreed to get them the full list of what we want to buy ASAP (ideally this week.
4. They are fully aware that a 14 day quote is a non-starter because we (or almost any customer) can't get the POs issued that quickly and they are working internally to get 30 day (or longer) commitments. To be fair to them, Dell is dealing with weekly supplier price increases and their suppliers won't guarantee prices beyond a short time.
5. They suggested that a few of us could possibly visit Round Rock for a fuller briefing on the future. The briefing would address pricing and availability for the next few years as well as how compute, storage, and networking were expected to evolve.
6. We also discussed bringing back the portal with ATLAS pricing and they were open to that. We pointed out having known pricing would ease our planning.
7. We told them about the LHC and ATLAS plans for run 4 and understand that we could need a lot of equipment starting in approximately 2029.
8. We told them that the EOL of the Dell OMSA server causes us difficulty and that we are stuck for RHEL10 which is not supported by OMSA. They are looking into how we might proceed.
9. They will check on the possibility to sell us ARM-based servers. We told them that the ATLAS software stack is fully validated on ARM.
10. We asked about other types of storage servers and it seemed to us that they really want us to stick to the R760xd2. It appeared that Dell wanted the simplest possible ask from our side so they could work with the current supply chain issues.

 

After the meeting Brian Teodecki sent us this message summarizing the discussion:

Dear ATLAS Team,

Thank you for taking the time to meet with us and share your plans, priorities, and challenges surrounding the upcoming infrastructure refresh across the ATLAS Tier 2 consortium.

We greatly appreciate the open discussion regarding your storage modernization strategy, procurement timelines, grant funding considerations, and the importance of maintaining a consistent experience across participating institutions. The insights provided by your team will help us better align our recommendations and support your objectives.

We understand the critical importance of balancing performance, scalability, budget constraints, and grant-driven procurement deadlines. We also recognize the value of delivering consistent configurations, predictable pricing, and timely execution across the consortium. Your feedback regarding lead times, OMSA transition planning, virtualization requirements, and future technology directions such as ARM-based computing was especially valuable.

Our team is already working through the requested follow-up items, including updated storage, networking, power infrastructure, and virtualization proposals. We will continue collaborating closely to provide the information and guidance needed to support your planning and decision-making process.

Dell Technologies values our long-standing partnership with the ATLAS research community and the important scientific work being conducted at your institutions. We are committed to helping you build a reliable, scalable, and cost-effective platform that enables future research success.

Thank you again for your partnership, candid feedback, and continued trust. We look forward to our next discussion and to supporting the consortium throughout this initiative.

Sincerely,

 

Brian Teodecki
Account Executive
Dell Tech

13:40 → 13:50 WBS 2.3.3 Heterogenous Integration and Operations

HIOPS

Convener: Rui Wang (Argonne National Laboratory (US))

Perlmutter: CPU allocation exhausted. Switched to overrun queue.

• Asking DOE if we could get top up for the last quarter
• Perlmutter is busy these days. Doug is playing with the Slurm worker settings (fewer jobs per worker and shorter time)
• Adding instructions for Perlmutter users (local job) under the ADC documentation page (to be merged: https://gitlab.cern.ch/atlas-us-hpc/atlas-computing-docs/-/blob/master/docs/analysis/ushpc_guild.md?ref_type=heads)

ALCF: DD renewal submitted for the next 6 months

13:40 HPC Operations

Speaker: Rui Wang (Argonne National Laboratory (US))

13:45 Integration of Complex Workflows on Heterogeneous Resources

Speaker: Doug Benjamin (Brookhaven National Laboratory (US))

13:50 → 14:10 WBS 2.3.4 Analysis Facilities

Convener: Wei Yang (SLAC National Accelerator Laboratory (US))

13:50 Analysis Facilities - BNL

Speaker: Qiulan Huang (Brookhaven National Laboratory (US))

• Continued JupyterHub Puppetization and automated the main deployment components, including configuration, authentication, custom spawner, proxy, and systemd services.

  • Started deployment validation on testnode and updated Python/package dependencies to align with the current production environment

• Discuss to make jupyterhub worker nodes also submit nodes for the shared pool

13:55 Analysis Facilities - SLAC

Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))

14:00 Analysis Facilities - Chicago

Speaker: Fengping Hu (University of Chicago (US))

Quarterly maintenance

• Quarterly maintenance was completed successfully.
• The MTU and DNS tuning changes appear to have improved cluster stability, and we're seeing fewer alerts overall

GPU allocation changes

• We reduced the minimum number of GPUs allocated to HTCondor from 9 to 3, so the default split is now 3 batch to 71 interactive.
• Batch can still scale up dynamically when demand is high.

 

14:10 → 14:30 WBS 2.3.5 Continuous Operations

Conveners: Ivan Glushkov (Brookhaven National Laboratory (US)), Ofer Rind (Brookhaven National Laboratory)

14:10 ADC Operations, US Cloud Operations: Site Issues, Tickets & ADC Ops News

Speaker: Kaushik De (University of Texas at Arlington (US))

• There have been multiple issues and blacklisting's affecting all sites during the past week. Note: if you see any new issue with your site, please check if it is a common issue first. Many of these issues were unexpected - therefore communication from central ADC was slow as they debugged the issue.
• Storms and power issues at multiple US sites, keeping site managers busy - otherwise smooth operations. 

14:15 Services DevOps

Speaker: Ilija Vukotic (University of Chicago (US))

14:20 Facility R&D

Speaker: Robert William Gardner Jr (University of Chicago (US))

14:25 Cybersecurity plan(s)

Speakers: Robert William Gardner Jr (University of Chicago (US)), Shigeki Misawa (Brookhaven National Laboratory (US))

14:30 → 14:40 AOB