US ATLAS Computing Facility (Possible Topical)

Wednesday 7 Oct 2026, 13:00 → 15:25 US/Eastern

Facilities Team Google Drive Folder

Because of ATLAS S&C week at CERN, we may have lighter attendance today!

Zoom information

Meeting ID:  993 2967 7148

Meeting password: 452400

Invite link:  https://umich.zoom.us/j/99329677148

 

 

13:00 → 13:05 WBS 2.3 Facility Management News

Speakers: Alexei Klimentov (Brookhaven National Laboratory (US)), Dr Shawn Mc Kee (University of Michigan (US))

This week is ATLAS S&C week at CERN https://indico.cern.ch/event/1712565/timetable/

Alexei and I need to summarize known scrubbing outcomes for WBS 2.3 (working on it)

  - This is especially needed so we can update our Milestones (many new WBS 2.3.x milestones were proposed during scrubbing)

  - We also need to verify all effort changes as part of this process

Dell is providing quotes the Tier-2s can use to estimate purchases from.  We need ALL Tier-2s to provide their estimates ASAP, so we can push for FINAL quotes (best pricing)

  - Current quotes are good till Nov 5

  - The switch pricing may not be very good right now.

  - No site can really order until we have those final quotes

  - Have all sites gotten their paperwork in?

HEPiX is coming up in 2 weeks: https://indico.cern.ch/event/1679145/

  - Today is the LAST day to submit abstracts or to register (we could use more people attending!)

US ATLAS–US CMS discussions at HEPiX

 

This HEPiX workshop, for the first time, will include a special series of focused USATLAS-USCMS discussions integrated into relevant tracks. The planned topics are:

• DC27: Opportunities for joint work and co-scheduled network data mini-challenges in the US. (Networking & Security track)
• Analysis facilities: How the experiments define, operate, and evolve analysis facilities, including the possible use of shared or federated resources. (Advances in Orchestrated Computing Platforms Topical Session)
• Operational AI: Large language models, Model Context Protocol data sources and tools, HEP-focused chatbots, benchmarking, and opportunities for collaboration. (Applied AI in Computing Center Infrastructures track)

ZOOM AI Notes Below

Quick recap

The meeting focused on operational updates and planning for upcoming events and infrastructure changes. Shawn reported that scrubbing outcomes and updated effort spreadsheets needed to be summarized and that milestones might need revision. He also discussed Dell quotes for tier twos, noting that total purchase estimates were needed and that some switch prices seemed high. The group discussed the upcoming HEPICS meeting, including a deadline confusion for abstracts and the plan for USATLAS and CMS joint sessions on analysis facilities. Brian provided an update on OSG26, stating it was expected to release on October 22nd. Gamboa reported that six new Atlas DCache pool servers were commissioned, providing nine petabytes of storage, and that work was underway to make REMAIN pool servers dual-homed. Ofer shared results from a transatlantic data challenge test, noting throughput between 200 and 400 gigabit per second and discussing some queued transfer issues. Rui gave an update on HPCs, mentioning the switch to the overrun queue and the status of ALCF allocations. Qiulan reported progress on JupyterHub publicization and configuration management at BNL. Fengping described server room reorganizations at Chicago and efforts to mitigate ServiceX app connection errors. The conversation ended with a discussion on the HEPICS analysis facility session, including the format of presentations and the potential for remote participation.

Next steps

Aidan

• Ask Jordan about including benchmarking slides in the HEPIX presentation and coordinate on Agentic analysis content.

Brian

• Submit abstracts for HEPIX by the end of this week.

Fengping

• Gather switch counters to confirm the cause of ServiceX connection errors.

Gamboa

• Coordinate with the networking group to complete the dual-home pool server fabric deployment by the end of November.

Ofer

• Fix the conflicting abstract submission deadline on the HEPIX website and clarify the deadline in communications.
• Begin setting up a FTS4 development instance at BNL with CERN's code, starting mid-October.

Qiulan

• Add a static landing page to the JupyterHub Puppet configuration.

Rui

• Continue developing the Harvard Surplus AI for Panda to submit to ALCF, including enabling Globus connectivity.

Shawn

• Finalize summarizing the scrubbing outcomes with Alexei, including updated effort spreadsheets and revisiting milestones, aiming to share with the group by the end of this week.
• Coordinate with Tier 2 sites to estimate and submit their Dell equipment purchases, ensuring all paperwork and funds are ready.
• Include the three EPICS topics in the meeting notes for today.
• Follow up with Way regarding the AGLT2 S3 local group disk storage issue.
• Coordinate with Andrew to prepare targeted questions for the Analysis Facility discussion at HEPIX.
• Add the AI Zoom notes to the top of the meeting notes after receiving them.

Summary

Atlas Software and Computing Updates

Shawn led a meeting to discuss several key topics including the ongoing Atlas software and computing week, the need to finalize and summarize scrubbing outcomes with Alexei, and the importance of revisiting milestones based on recent changes. The team discussed receiving updated effort spreadsheets and quotes from Dell for tier two equipment purchases, with Shawn noting concerns about higher switch prices compared to last year. Shawn also reminded attendees about the upcoming EPICS event in two weeks, emphasizing the deadline for abstract submissions and registrations.

HEPIC Abstract Registration Deadline

The team discussed the HEPIC abstract registration deadline, which was confirmed to be today, though there was initial confusion about different dates on different pages. Ofer mentioned sending an email to the HEPIX list and encouraged submissions of abstracts, site reports, and AI sessions, particularly highlighting the opportunity for USATLAS CMS joint sessions. Brian announced his registration for HEPIC and mentioned he might submit abstracts by the end of the week, while also updating that OSG26 is scheduled for release on October 22nd, with OSG24 reaching end-of-life on the same date.

Data Storage and Network Updates

Gamboa reported on operations updates, including the commissioning of six new Atlas DCache pool servers providing nine petabytes of storage, with six of the pools supporting DMC workload for third-party copies. The networking team is working on deploying fabric to transform remaining pool servers into dual-home installations, expected to be completed by end of November. Ofer discussed a recent transatlantic data challenge test between CERN, BNL, and Fermilab, which achieved throughput of 200-400 gigabit per second using a combination of old door methods and dual-homed pools, with FTS4 being used for the test.

FTS4 Development and Data Challenge

Gamboa reported on an analysis of queued transfers where some transfers got stuck in a state where data could not be moved from FTS, but clarified this issue was not unique to VNL and affected other sites like IN2P as well. CERN agreed to provide FTS4 code to BNL for setting up a development instance to test before DC27, with mid-October planned for setup. Shawn expressed concerns about running FTS4 in production for DC27 and suggested the possibility of delaying the Data Challenge by a month or two, though acknowledged the challenges involved in making such changes.

HPC and Infrastructure Updates

Gamboa reported updated pledge requests from the collaboration showing increased costs, particularly for tape storage which saw significant increases. The team discussed recent database issues that drained staff resources earlier in the week. Rui provided updates on HPC operations, noting a switch to the overrun queue with adjustments to node workers, and reported on ARCF infrastructure including renewed DDR location and current system allocations.

Harvard Surplus AI Development Progress

Rui explained that they are developing Harvard Surplus AI for Panda to submit to ALCF, but the system is not yet finished, particularly regarding Globus connection to BIRD cache for production data transfers. Qiulan reported significant progress on Jupyter Hub deployment, completing publicization of all Jupyter Hub configurations with automated management through Puppet and Red Hat 9 OS kernel. The team plans to further improve the system by adding a static landing page and making JupyterHub work nodes submit to the shared pool including GPU nodes.

Server Room Reorganization and Networking

Fengping discussed server room reorganization plans in Chicago involving the consolidation of AF nodes onto Switch A17 and N20 to prepare for a network refresh updating uplinks to 100 gigabit. He also addressed ServiceX app connection errors during heavy load testing, which were attributed to switch congestion, and reported that developers increased the connection timeout from 0.5 to 1.5 seconds as a mitigation. Ofer inquired about future network throughput testing, but Fengping indicated those tests were not currently planned. Ofer then provided an update on continuous integration and operations, noting quiet activity due to SNC week and mentioning a mass Q blacklisting event caused by a Panda database file system issue.

AGL Tier 2 Transfer Issues

The team discussed ongoing inbound transfer efficiency problems at AGL Tier 2, which Shawn attributed to issues with the S3 local group disk that Way manages. Shawn reported implementing fixes on the backend Ceph system but had not received feedback from Way about the status of the VM. Gamboa informed the group about a new DCATCH minor release (11.2.08) that affects systems running on version 11, noting that while there were some issues found with the movers on the NFS, it was considered a minor update. The conversation ended with a discussion about planning for an Analysis Facility presentation at HEPICS, which Ofer explained would be co-organized with others and tied to a broader session on orchestrated platforms, including discussions about Kubernetes advances and inference servers.

Analysis Facilities Session Planning

The team discussed plans for an upcoming analysis facilities session, with Shawn suggesting a 15-minute presentation from Aidan followed by 30 minutes of discussion. Aidan agreed to focus on the UChicago Analysis Facility and planned to include discussion of data access methods and integration with national cyber infrastructure. The group also addressed logistics, confirming that while remote access would be available for presentations via Zoom, in-person attendance was preferred for those making presentations.

 

13:05 → 13:10 OSG-LHC

Speakers: Brian Hua Lin (University of Wisconsin), Matyas Selmeci

• Aiming for OSG 26 release + OSG 24 EOL on Oct 22
• BrianL to attend HEPiX

13:10 → 13:30 WBS 2.3.1: Tier1 Center

Convener: Alexei Klimentov (Brookhaven National Laboratory (US))

13:10 Tier-1 Infrastructure

Speaker: Jason Smith

13:15 Compute Farm

Speaker: Thomas Eric Smith (Brookhaven National Laboratory (US))

13:20 Storage

Speakers: Carlos Fernando Gamboa (Department of Physics-Brookhaven National Laboratory (BNL)-Unkno), Carlos Fernando Gamboa (Brookhaven National Laboratory (US))

• No major issues in operations for dCache or HPSS last week 
  Commissioned six new ATLAS dCache pool servers (dc295–dc300), adding approximately 9 PB of storage capacity to production.
  Selected pools were also configured to support third-party copy (TPC) workflows for both disk and tape storage. Their dual-homed LAN/WAN connectivity allows eligible transfers to communicate directly with the pools, expanding the Storage Element’s SciDMZ transfer capacity and reducing reliance on the dCache door servers and their DMZ doors NVMe buffers.
• Internal coordination among SCDF groups to review DUAL home deployment for pools, related network hardware in progress to be acquired
• ENDIT provider updated for better error log handling

13:25 Tier1 Operations and Monitoring

Speaker: Ofer Rind (Brookhaven National Laboratory)

• Transatlantic Data Challenge last week (gdoc) (MM channel)
  • BNL around 200 to 400 Gbps
    • Some remaining questions about impact of queued transfers
  • FTS4 used.
• BNL will get a dev instance of FTS4 in mid-Oct

13:30 → 13:40 WBS 2.3.2 Tier2 Centers

Updates on US Tier-2 centers

Conveners: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

13:40 → 13:50 WBS 2.3.3 Heterogenous Integration and Operations

HIOPS

Convener: Rui Wang (Argonne National Laboratory (US))

13:40 HPC Operations

Speaker: Rui Wang (Argonne National Laboratory (US))

Perlmutter: <900 GPU node hours remain

• Doug changed the workers to 2 nodes, 6 hours to help them get queued faster via overrun queue

JobID             User   NNodes     ReqMem     MaxRSS     AveRSS        QOS ElapsedRaw  Partition      State  Timelimit    Planned 
------------ --------- -------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- 
58826554       usatlas        2    975604M                          overrun       6287 regular_m+  COMPLETED   02:10:00 8-23:29:22

ALCF: DD allocation renewed for 6 months

• Crux 10K, Polaris 5K, Sophia 2K, Eagle 100TB

13:45 Integration of Complex Workflows on Heterogeneous Resources

Speaker: Doug Benjamin (Brookhaven National Laboratory (US))

13:50 → 14:10 WBS 2.3.4 Analysis Facilities

Convener: Wei Yang (SLAC National Accelerator Laboratory (US))

13:50 Analysis Facilities - BNL

Speaker: Qiulan Huang (Brookhaven National Laboratory (US))

• Updates on the Jupyterhub puppet
  • Completed the Puppetization of the JupyterHub deployment, establishing an automated and standardized configuration management process for JupyterHub nodes after testing and debug to fix a number of issues.
  • Managed to make JupyterHub running on the testnode, with the deployment and configuration managed by Puppet.
  • Installed on RHEL9
  • Further improvements will be added if needed 
• Our farm group are going to make jupyterhub worker nodes (including GPU nodes) also submit nodes for the shared pool

13:55 Analysis Facilities - SLAC

Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))

14:00 Analysis Facilities - Chicago

Speaker: Fengping Hu (University of Chicago (US))

Server room reorganization

• Planning to swap AF workers and MWT2 workers between the two involved switches, consolidating all AF hosts onto switches A17 and A20. This prepares for the network refresh, which will upgrade the uplinks to N×100 Gbps.

ServiceX app connection errors

• Errors were reported while the cluster was under heavy load (~17 Gbps of Ceph reads).
• Testing showed some connections taking ~1.02 s instead of the usual ~1 ms. This is consistent with a dropped SYN/SYN-ACK and the 1 s TCP retransmit timeout, so switch congestion is suspected.
• Mitigation: the ServiceX connection timeout was raised from 0.5 s to 1.5 s, so a connection can now survive a single SYN/SYN-ACK retransmit.
• Gathering switch counters to confirm the root cause.

14:10 → 14:30 WBS 2.3.5 Continuous Operations

Conveners: Ivan Glushkov (Brookhaven National Laboratory (US)), Ofer Rind (Brookhaven National Laboratory)

• Daily ops meetings are virtual during ATLAS S&C Week
• Mass queue blacklisting event yesterday due to PanDA DB issue - filesystem file size limit reached (OTG0163574)
• Low inbound transfer efficiency at AGLT2 for the past 48 hours?

14:10 ADC Operations, US Cloud Operations: Site Issues, Tickets & ADC Ops News

Speaker: Kaushik De (University of Texas at Arlington (US))

14:15 Services DevOps

Speaker: Ilija Vukotic (University of Chicago (US))

14:20 Facility R&D

Speaker: Robert William Gardner Jr (University of Chicago (US))

14:25 Cybersecurity plan(s)

Speakers: Robert William Gardner Jr (University of Chicago (US)), Shigeki Misawa (Brookhaven National Laboratory (US))

14:30 → 14:40 AOB