US ATLAS Computing Facility (Possible Topical)

Wednesday 12 Aug 2026, 13:00 → 15:25 US/Eastern

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  993 2967 7148

Meeting password: 452400

Invite link:  https://umich.zoom.us/j/99329677148

 

 

13:00 → 13:05 WBS 2.3 Facility Management News

Speakers: Alexei Klimentov (Brookhaven National Laboratory (US)), Dr Shawn Mc Kee (University of Michigan (US))

Upcoming important activities for the facility:

• Approve and finalize new milestones
  • See WBS 2.3 copy for existing milestones
• WLCG mini-challenge planning
• End-of-CA equipment purchases
• Getting summary of all scrubbing outcomes impacting WBS 2.3

 

We have some scrubbing results to share for WBS 2.3.2 (Tier-2s).   See the table below.  Important to note are that each Tier-2 will receive exactly $500K for FY27 and there will be no hardware procurement:

 

WBS	End of CA Material (approximate)	FY27 Cost (old CA)	FY27 Cost (new CA)	FY27 COLA	FY27 Trav/Mat	Oct-Jan Funding	Feb-Apr Funding	May-Sept Funding	Notes
2.3.2.1: AGLT2	2000	490.6	500.0	included	included	163.520	125.000	208.333	No more hardware in FY27
2.3.2.2: MWT2	1600	594.6	500.0	included	included	198.200	125.000	208.333	No more hardware in FY27
2.3.2.3: NET2	2500	366.9	500.0	included	included	122.300	125.000	208.333	No more hardware in FY27
2.3.2.4: SWT2	2500	317.6	500.0	included	included	105.867	125.000	208.333	No more hardware in FY27

 

 

Quick recap

The meeting covered updates and discussions on facility operations, budget impacts from scrubbing, and upcoming activities. Shawn shared that each Tier 2 would receive $500K for FY27 with no hardware procurement, and highlighted WLCG mini challenges and a planned dry run for DC27. Ofer announced the opening of HEPiX registration and encouraged facility attendance, especially for sessions on orchestrated platforms and facilities planning. Brian provided updates on OSG software releases and ongoing troubleshooting for memory leaks in XRootD. Thomas and Gamboa reported stable operations at the Tier 1, with minor issues like a brief dip in idle jobs and ongoing storage upgrades. Rui updated on HPC resources and technical setups, while Qiulan and Fengping shared updates on analysis facility support and service improvements. Ofer noted the migration of tape-related transfers to FTS4 and issues with transfer failures at some sites. The conversation ended with no additional business.

Next steps

Brian

• Follow up on the email thread regarding XRootD monitoring with the Southwest Tier 2.

Douglas

• Send the relevant snippet of the WLCG Technical Coordination Board document section to Brian for feedback on HPCs integration.

Fengping

• Update the meeting notes with a link to the USATLAS workshop that used the Power AF services.

Gamboa

• Provide feedback to Mark regarding the current and future (5-year horizon) network connectivity (ESNet links) for the Tier 1.

Shawn

• Follow up with Andrew Mello and others to coordinate a possible joint USATLAS-US-CMS meeting during HEPIX, targeting Friday afternoon.

Summary

FY27 Facility Funding and Activities

Shawn outlined upcoming facility activities including scrubbing milestones, WLCG mini challenges, and plans for a DC27 dry run in December. He announced that each Tier 2 site will receive $500K for FY27, which must cover all expenses including labor, travel, and supplies, with no hardware procurement allowed.

Budget and Project Updates Meeting

The team discussed pending budget information for tier one, with potential support requested for VO admin and USATLAS cybersecurity at 0.2-0.25 FTE levels. Ofer announced that HEPIC registration opened this week with topical sessions on orchestrated platforms and hardware market shifts, and mentioned a potential joint OTF session related to the WLCT technical roadmap that Ale is working on.

CMS Attendance Event Planning

The group discussed organizing a CMS attendance event, with Shawn confirming he had already registered and suggesting Friday afternoon as a potential time slot during the 4.5-day HEPIX meeting. Shawn mentioned that the WLCG roadmap delivery has been delayed to early 2027 due to Atlas requests, though a draft will be available by the time of the meeting. Ofer suggested following up with CMS contacts including Andrew Mello to coordinate the event and potentially focus on the facilities chapter of the roadmap draft.

Software Release Updates and Planning

Brian provided updates on several releases, including an HT Condor release fixing security bugs and an upcoming XRAD 6.1.1 release that will be built and added to repositories. The team is preparing for OSG, HDCSS, and Pelican 26 releases targeted for end of September, which will end OSG 24 support. Douglas raised a question about OSG TCB documentation requiring feedback on HPC infrastructure integration, and agreed to send relevant details to Brian for review.

Compute and Storage Operations Update

Thomas reported smooth operations for compute with no major hardware or software changes, except for a brief issue on August 10th where idle jobs nearly ran out but automatically recovered without intervention. The planned power intervention downtime for August 14th-16th has been postponed and will be rescheduled. Gamboa discussed storage operations, noting smooth cache and HPSS operations, and mentioned ongoing work on new pool servers awaiting JBODs expected in two months. A tentative decache upgrade is scheduled for September 16th, though this needs confirmation due to the postponed power intervention.

Network Connectivity and Bandwidth Planning

The team discussed network connectivity and bandwidth requirements for the next five years. Shawn explained that while current bandwidth for BNL appears sufficient for ALICE's needs, any future upgrades would depend on ESNet's plans, which are typically considered 5-7 years ahead. The group also addressed a planned facility outage originally scheduled for Saturday, which may be postponed to September and potentially combined with DCache and other updates. Ofer mentioned that WSD TAPE account counting numbers were being recalculated and the automated recording system had been restored.

HPC Resources and Updates

The team discussed updates on HPC resources, including the status of GPU node hours and Vanish service deployment pending security team approval. Rui reported on ALCF account activation and credential installation in a virtual machine. Qiulan provided updates on BRL facility, including support during training week and development of puppet code for Jupyter, with plans to begin testing the following week.

Chat Service Migration Updates

Fengping reported improvements to the chat inference service, including local bearer token validation and ongoing authentication service migration from the Tempest cluster to the AF cluster ahead of scheduled data center maintenance. Ofer provided updates on CI/Operations, noting the migration of tape-related transfers to FTS4 and progress at various Tier 2 sites, though some transfer failures remain. The next meeting is scheduled for August 26th.

13:05 → 13:10 OSG-LHC

Speakers: Brian Hua Lin (University of Wisconsin), Matyas Selmeci

1. Release
   1. HTCondor fix incoming for broken FS auth + TCP_FORWARDING_HOST
   2. XRootD 6.1.0 has looked good in our testing but Andy has suggested that we wait to release 6.1.1
   3. We're gearing up for OSG / HTCSS / Pelican 26. Targeting end of September, which will EOL 24)
2. CMS has reported XRootD redirector memory leaks
3. SWT2: XRootD monitoring question

13:10 → 13:30 WBS 2.3.1: Tier1 Center

Convener: Alexei Klimentov (Brookhaven National Laboratory (US))

13:10 Tier-1 Infrastructure

Speaker: Jason Smith

13:15 Compute Farm

Speaker: Thomas Eric Smith (Brookhaven National Laboratory (US))

No major changes to report for hardware/software

Operations have been very smooth - there was brief moment (~30 minutes) on 10 August where we nearly ran out of idle jobs, but it bounced back without intervention on our side

Power intervention downtime 14-16 August has been postponed, new date TBD. Assume regular T1/T3 AF operations this weekend

13:20 Storage

Speakers: Carlos Fernando Gamboa (Brookhaven National Laboratory (US)), Carlos Fernando Gamboa (Department of Physics-Brookhaven National Laboratory (BNL)-Unkno)

• No major issues to report on dCache and HPSS operations

• Commissioning of 8 new pool servers is in progress

  • Procurement of the associated JBODs is underway expected to arrive in 2 months

• Tentative date for dCache upgrade 11.2.X  (09/16/26;9AM-5PM) TBC
  • candidate 11.2.5 --> 11.2.6 released on Monday this week
• SCDF Power intervention for 08/14-16 is cancelled
  • dCache released services will resume as usual

13:25 Tier1 Operations and Monitoring

Speaker: Ofer Rind (Brookhaven National Laboratory)

• Facility outage planned for this Saturday has been postponed, date TBD.
  • This will also postpone the planned HTCondor and OS updates.  Hope to schedule those concurrent with rescheduled electrical outage.
  • dCache update currently planned for around Sept. 15th, may try to schedule that with the maintenance as well, if possible.
• Recalculating WLCG tape accounting numbers for June and July
  • Automated reporting has been restored

13:30 → 13:40 WBS 2.3.2 Tier2 Centers

Updates on US Tier-2 centers

Conveners: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

13:40 → 13:50 WBS 2.3.3 Heterogenous Integration and Operations

HIOPS

Convener: Rui Wang (Argonne National Laboratory (US))

13:40 HPC Operations

Speaker: Rui Wang (Argonne National Laboratory (US))

Perlmutter: ~155K (18%) / 15K (20%) CPU/GPU node hours remain.

• No news on additional GPU hours yet

TACC: update on Vanish from support -- the new system has been set up for a bit. We are just waiting on internal security scans basically. After that we can release them to the world. At that point we test further. 

ALCF: Wen's ALCF account has been activated. He got a VM with production role credentials installed with Harvester to work with

 

13:45 Integration of Complex Workflows on Heterogeneous Resources

Speaker: Doug Benjamin (Brookhaven National Laboratory (US))

13:50 → 14:10 WBS 2.3.4 Analysis Facilities

Convener: Wei Yang (SLAC National Accelerator Laboratory (US))

13:50 Analysis Facilities - BNL

Speaker: Qiulan Huang (Brookhaven National Laboratory (US))

• Provide smooth support throughout the training week.
• Continue work on the data access analysis, focusing on /pnfs/usatlas.bnl.gov/users/…
  • Results show the data access in /pnfs/usatlas.bnl.gov/users/... is not active.
• Continue developing the Puppet code for Jupyter; testing will begin next week

13:55 Analysis Facilities - SLAC

Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))

14:00 Analysis Facilities - Chicago

Speaker: Fengping Hu (University of Chicago (US))

• Triton Inference Service improvements
  • Bearer token authentication is now validated locally at the ingress level instead of being forwarded to the OAuth proxy.
  • This change has significantly improved inference throughput.
• Authentication service migration
  • The Keycloak instance providing authentication for several AF applications, including ServiceX and Coffea-Casa, is currently hosted on the Tempest cluster.
  • The data center hosting Tempest is scheduled for maintenance on August 18–19.
  • We are using this opportunity to migrate the Keycloak service to the AF cluster to avoid service disruption.

• US ATLAS / IRIS-HEP Analysis Software Training Event 2026

  • Successfully supported the 2026 training event, including AF and SSL access for workshop participants.
• Quarterly maintenance
  • The next quarterly AF maintenance is scheduled for August 31

14:10 → 14:30 WBS 2.3.5 Continuous Operations

Conveners: Ivan Glushkov (Brookhaven National Laboratory (US)), Ofer Rind (Brookhaven National Laboratory)

14:10 ADC Operations, US Cloud Operations: Site Issues, Tickets & ADC Ops News

Speaker: Kaushik De (University of Texas at Arlington (US))

• Tape related transfers between selected sites migrated to FTS4 at CERN today (as of now, BNL has not been migrated)
• Backoff errors at NET2 have been reduced by removing the dense nodes, resulting in a temporary reduction of job slots while disks are being replaced
• SWT2 k8s cluster nodes have been migrated to the standard queue
• Multiple HC exclusions at SLAC due to transfer failures
  • Overall, a number of destination sites (non-US) with ongoing transfer failures

14:15 Services DevOps

Speaker: Ilija Vukotic (University of Chicago (US))

14:20 Facility R&D

Speaker: Robert William Gardner Jr (University of Chicago (US))

14:25 Cybersecurity plan(s)

Speakers: Robert William Gardner Jr (University of Chicago (US)), Shigeki Misawa (Brookhaven National Laboratory (US))

14:30 → 14:40 AOB