US ATLAS Computing Facility (Possible Topical)

Wednesday 26 Aug 2026, 13:00 → 15:25 US/Eastern

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  993 2967 7148

Meeting password: 452400

Invite link:  https://umich.zoom.us/j/99329677148

 

 

13:00 → 13:05 WBS 2.3 Facility Management News

Speakers: Alexei Klimentov (Brookhaven National Laboratory (US)), Dr Shawn Mc Kee (University of Michigan (US))

Not yet able to spend end-of-CA funds

Still waiting on comprehensive scrubbing results for WBS 2.3

Today we have a topical report on optimizing the equipment configuration quotes

Other items?

 

Quick recap

The meeting focused on WBS 2.3 activities, with discussions covering OSG 26 development progress, a critical security vulnerability identified in the PerfSonar service at Southwest Tier 2, and equipment procurement planning. Brian reported on OSG 26 foundations being laid with Koji RPM build infrastructure and Yum repositories, while XRootD 5.9.7 testing revealed memory issues on CMS redirectors. Kaushik raised urgent security concerns about unauthorized code injection and network mapping through PerfSonar, leading to detailed discussions about reporting procedures and potential fixes including switching to containerized versions. Frederick presented findings from the FY26 Procurement Working Group on optimizing equipment configurations, particularly focusing on storage server options and Dell pricing strategies, with emphasis on combining site requirements for better discounts. The conversation ended with updates from various Tier 1 and Tier 2 sites on operations, maintenance, and ongoing projects.

Next steps

Frederick

• Send the link and information for the next FY26 Procurement Working Group meeting to a general list, including non-Tier 2 participants.
• Add Judith to the mailing list for the FY26 Procurement Working Group.
• Work with the working group to define server configurations (bins) and gather total spending estimates from Tier 2 sites for Dell negotiations.
• Update the funding model for Tier 2s, incorporating retirements and needs, for the next CA grant application.

Ilija

• Debug the issue with BNL's XCache not serving files through the cache, in coordination with Andy Hamczewski.

Kaushik

• Provide detailed information about the PerfSonar security vulnerability (code injection, internal network exposure) to Shawn for reporting to the PerfSonar team.
• Respond to the university's security concerns, informing them of the switch to the containerized PerfSonar version and that the issue is being addressed.

Kevin

• Discuss offline with Thomas the possibility of offering a test access point to the shared pool for ATLAS.

Ofer

• Confirm the version (11.2.6) for the upcoming dCache upgrade with Eduardo.

Shawn

• Open a thread with the PerfSonar team and relevant security contacts (e.g., OSG Security, Rob, Shigeki) to report and address the identified PerfSonar vulnerability.
• Reply to Kaushik's email with the recommendation to switch to the containerized version of PerfSonar, including instructions.
• Follow up with the University of Michigan administration regarding grant rules for delayed equipment delivery and share the findings with the group.

Summary

OSG 26 Development Progress Update

The meeting covered updates on OSG 26 development, with Brian reporting progress on Koji RPM build infrastructure and OSG Yum repositories, along with plans for testing new major versions in September. Brian also discussed XRootD 5.97 testing, which showed increased CPU utilization at UNL, and mentioned that XRootD 6.0 builds are in testing for potential inclusion in OSG 26. The meeting was interrupted when Kaushik raised an urgent security issue involving executable injection, though the details were not fully discussed before the transcript ended.

PerfSonar Security Vulnerability Discussion

The team discussed security issues identified with PerfSonar, with Kaushik emphasizing the need for urgent action on three security vulnerabilities that were reported. Shawn clarified that while the issues described in the email were by design rather than security flaws, they would engage with the PerfSonar team to properly evaluate and address the concerns. The group agreed to have a follow-up meeting immediately after the current meeting to discuss the details further, after which they would involve the appropriate OSG security teams and PerfSonar developers to handle the vulnerabilities appropriately.

Rucio Support and Procurement Updates

The team discussed Rucio's plan to drop support for specific environment variables (ATLAS site name and OSG site name) in future versions, with Ivan clarifying that sites should use the standard "site name" variable instead. Frederick presented a report from the FY26 Procurement Working Group regarding Dell negotiations, explaining that the working group is developing questions for Dell's sales team to determine optimal server and network switch configurations for better pricing. Kaushik emphasized the importance of gathering total spending across all sites before approaching Dell, suggesting that including network switch purchases in the negotiation could lead to better overall pricing.

Equipment Purchase Optimization Strategies

Frederick led a discussion on optimizing equipment purchases, emphasizing the focus on storage rather than compute due to rising memory prices. He outlined strategies to combine site requests, minimize server configurations, and consider warranty options, while noting challenges with Dell's pro support costs. The group debated storage configurations, with Eduardo suggesting ZFS as an alternative to RAID 6, and Judith highlighting the additional considerations required for implementing ZFS, including memory requirements and monitoring infrastructure. Kevin emphasized the importance of memory capacity for storage super servers, while Shawn encouraged further detailed discussions to be moved to the working group.

Storage System Options Discussion

The team discussed storage system options, with Frederick presenting two potential approaches including traditional Dell systems and newer higher-performance alternatives that Dell currently only sells in larger configurations. The group agreed to consult with Dell about potentially accessing these higher-performance systems through combined facility purchases, while planning to request configurations with Xenon Silver CPUs and 128GB or 256GB memory options. Thomas provided updates on Tier 1 operations, including a delay in the rolling upgrade due to S-Phoenix work and an increase in maximum concurrent uploads from 20 to 100 per user, while Ofer reported that network configuration updates would allow for rebuilding of FTS dev and Persona hosts, and confirmed that the decache upgrade is scheduled for September 14th.

Operations and Technical Updates Meeting

The team discussed recent operations and updates across multiple areas. Frederick reported smooth operations but noted uncertainty about funding details, which he plans to follow up on with John Hobbs. Shawn provided updates on projections and models, clarifying that while the model is still needed, the funding numbers for Tier 2s have already been provided. Rui reported receiving 75k CPU node hours for HPC parameter work but no additional GPU node hours, and mentioned progress on ARCF API development and combining workers across different sites. Fengping outlined upcoming quality maintenance activities including firmware and OS updates, deployment of an MCP portal, and Gateway API implementation. Ilija reported several technical issues including XCache connection problems at BNL and VP service security updates at CERN, with ongoing work on Series 6 changes and AI bot updates. The conversation ended with a detailed discussion about security concerns raised by the university regarding PerfSonar, where it was decided to recommend switching to the containerized version to address security vulnerabilities and data exposure concerns.

13:05 → 13:10 OSG-LHC

Speakers: Brian Hua Lin (University of Wisconsin), Matyas Selmeci

• OSG 26
  • We're starting to lay down Koji build infrastructure and repos
  • Aiming to start testing in early Sep and release in late Sep
• XRootD 5.9.7 in testing but UNL saw large CPU increases in their redirector

13:10 → 13:30 Topical: USATLAS Quote Configure WG Report

Fred will provide an update on the current work for defining end-of-CA quotes.

Speaker: Fred Luehring (Indiana University (US))

FY26 Procurement WG Report.pdf

13:30 → 13:50 WBS 2.3.1: Tier1 Center

Convener: Alexei Klimentov (Brookhaven National Laboratory (US))

13:30 Tier-1 Infrastructure

Speaker: Jason Smith

13:35 Compute Farm

Speaker: Thomas Eric Smith (Brookhaven National Laboratory (US))

• No urgent errors to report
• rolling upgrade was supposed to have started Monday, but was bumped by more urgent sPHENIX work
• Increased the MAX_CONCURRENT_UPLOADS_PER_USER knob (value was 20, now is 100) on AT1 and AT3/AF submit nodes. This was increased based on findings from the sPHENIX work

13:40 Storage

Speakers: Carlos Fernando Gamboa (Brookhaven National Laboratory (US)), Carlos Fernando Gamboa (Department of Physics-Brookhaven National Laboratory (BNL)-Unkno)

13:45 Tier1 Operations and Monitoring

Speaker: Ofer Rind (Brookhaven National Laboratory)

• Hiro will be rebuilding FTS dev and PerfSONAR hosts
• HTCondor-CE upgrades this week
• dCache upgrade schedule for Monday Sept. 14th after the electrical maintenance downtime

13:50 → 14:00 WBS 2.3.2 Tier2 Centers

Updates on US Tier-2 centers

Conveners: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

14:00 → 14:10 WBS 2.3.3 Heterogenous Integration and Operations

HIOPS

Convener: Rui Wang (Argonne National Laboratory (US))

Perlmutter: 75k CPU node hour added. No luck on the GPU side

ALCF: maintenance last week and this Monday

• All ALCF users can now submit jobs to Polaris and Crux through the ALCF IRI API
• Working on the PBS template and wrapper for harvester worker submission test_submitter for IRI testing
  • combine the current individual ones per PanDA queue into a common one, supports both setupAtlas and CVMFSExec
• Wen has successfully submitted a job via IRI to Crux. He needs permission to write into the BNL dCache area for data transfer test using IRI

14:00 HPC Operations

Speaker: Rui Wang (Argonne National Laboratory (US))

14:05 Integration of Complex Workflows on Heterogeneous Resources

Speaker: Doug Benjamin (Brookhaven National Laboratory (US))

14:10 → 14:30 WBS 2.3.4 Analysis Facilities

Convener: Wei Yang (SLAC National Accelerator Laboratory (US))

14:10 Analysis Facilities - BNL

Speaker: Qiulan Huang (Brookhaven National Laboratory (US))

14:15 Analysis Facilities - SLAC

Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))

14:20 Analysis Facilities - Chicago

Speaker: Fengping Hu (University of Chicago (US))

• Quarterly Maintenance: The next quarterly AF maintenance is scheduled for August 31 and will include firmware, OS, Kubernetes, HTCondor, and CVMFS updates. We will also update the Calico MTU from 1500 to 8500.
• AF MCP Portal: The AF MCP Portal is currently in alpha/beta and is now linked from the AF portal. It provides AI agents with access to facility services by brokering tool access and credential management.
• Gateway API: Gateway API has been deployed with the Envoy Gateway controller, providing dual-stack networking and wildcard TLS certificates for the af.uchicago.edu domain. Applications can now migrate from traditional Ingress resources to HTTPRoute resources.

14:30 → 14:50 WBS 2.3.5 Continuous Operations

Conveners: Ivan Glushkov (Brookhaven National Laboratory (US)), Ofer Rind (Brookhaven National Laboratory)

14:30 ADC Operations, US Cloud Operations: Site Issues, Tickets & ADC Ops News

Speaker: Kaushik De (University of Texas at Arlington (US))

• WLCG OTF #12 was this week.  Excellent talks by Rob, Verena, et.al.

14:35 Services DevOps

Speaker: Ilija Vukotic (University of Chicago (US))

• XCaches
  • a few issues with monitoring (test files at BNL not accessible)
  • missing test results from one of UK sites
• VP service
  • needed a security update
  • used is as an opportunity to fully update
  • added more resources to its CERN Openstack project, created a new "dev" k8s cluster, created new deployment with new way to provide ingress (gateway)
  • all working but waiting on CERN IT to finish their security tests before opening firewall putting it in use.
• Conditions delivery
  • Frontiers worked fine
  • Varnishes worked fine
  • AI bot monitoring it worked fine
• ServiceX/ServiceY
  • ServiceX logging and ES templates updated
  • ServiceY images brought up to parity with ServiceX
• AI work
  • infrastructure running on our Sparks nodes updated
  • new bots added:
    • PIC operations for Andrieu
    • score-bot for Natalia and the benchmarking team

14:40 Facility R&D

Speaker: Robert William Gardner Jr (University of Chicago (US))

14:45 Cybersecurity plan(s)

Speakers: Robert William Gardner Jr (University of Chicago (US)), Shigeki Misawa (Brookhaven National Laboratory (US))

14:50 → 15:00 AOB