US ATLAS Tier 2 Technical

Wednesday 16 Sept 2026, 11:00 → 12:00 US/Eastern

Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), Shawn Mc Kee (University of Michigan (US))

Meeting to discuss technical issues at the US ATLAS Tier 2 site. The primary audience is the US Tier 2 site administrators but anyone interested is welcome to attend.

11:00 → 11:10 Introduction

Speakers: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

Quick links:

• Capacity spreadsheet
• Services spreadsheet 

General news

• Initial quote for storage servers from Dell received. 
  • $59,370.00 for 28x24T disks
  • No solid state NVMe drives
  • Approximately $100/useful TB (100% increase compared to a few years ago)
• Sites should confirm that this configuration can be used or if there are any showstoppers (either technical or strategic).
• Delivery is expected in February if we place order now
  • But no funds available yet
  • Management is investigating possible 3 months extension for purchase (until Apr/2026), but nothing is confirmed yet.

Upcoming meetings:

• 57th LHCOPN-LHCONE meeting [Oct 1st -- 2nd, Naples]
• #85 ATLAS S&C week  [Oct 5th -- 9th, CERN]
• HEPiX Fall 2026 Workshop [Oct 19th -- 23rd, Nebraska] -- Any updates on the USATLAS/USCMS facility discussion?
• WLCG/HSF Workshop 2026 [Nov 2nd -- 6th, Bologna]
• SC26 [Nov 16nd -- 21st, Chicago]

Open tickets:

• ggus:1003665 TW-FTT: Anonymous access to ATLAS data on TW-LCG2_DATADISK
• ggus:1003053 SWT2/CPB: IGTF CRLs & fetch-crl SHA1 signature validation

11:10 → 11:20 TW-FTT

Speakers: Eric Yen, Felix.hung-te Lee (Academia Sinica (TW)), Yi-Ru Chen (Academia Sinica (TW))

• Scheduled downtime:  site will be shutdown from 2026-09-19 10:00 a.m. to 2026-09-24 10:00 a.m. (UTC) because of high-voltage switchgear maintenance.
• ggus:1003665 TW-FTT: Anonymous access to ATLAS data on TW-LCG2_DATADISK
  • still investigating which configuration setting is wrong.

11:20 → 11:30 AGLT2

Speakers: Daniel Hayden (Michigan State University (US)), Philippe Laurens (Michigan State University (US)), Shawn Mc Kee (University of Michigan (US)), Dr Wendy Wu (University of Michigan)

  Smooth running over last 2 weeks

  One dCache pool node (R760xD2 with big pools) now on RHEL 10 and in production.

  built 2 work nodes (1 vm with 2 cores, 1 bare metal Dell work node) with RHEL10, have been running ATLAS jobs for over a week smoothly.

Dell OMSA software kit is completely out on RHEL10. 

11:30 → 11:40 MWT2

Speakers: Aidan Rosberg (Indiana University (US)), David Jordan (University of Chicago (US)), Farnaz Golnaraghi (University of Chicago (US)), Fengping Hu (University of Chicago (US)), Fred Luehring (Indiana University (US)), Judith Lorraine Stephen (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

• GGUS-Ticket-ID: #1003894: XCache ticket due to a certificate issue. Resolved by Ilija
• Otherwise smooth running

11:40 → 11:50 NET2

Speakers: Eduardo Bach (University of Massachusetts (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), William Axel Leight (University of Massachusetts Amherst)

Had another surge of backoff errors, this time traced them to an issue with the container interface process running out of memory which causes all containers to be killed.  Increasing the available resources for that process seems to have resolved the issue.

Also saw a recurrence of merge jobs failing due to insufficient storage, increased the safety threshold from 120% to 160% to try to prevent those.

11:50 → 12:00 SWT2

Speakers: Andrey Zarochentsev (University of Texas at Arlington (US)), Horst Severini (University of Oklahoma (US)), Kaushik De (University of Texas at Arlington (US)), Mark Sosebee (University of Texas at Arlington (US)), Zachary Thomas Booth (University of Texas at Arlington (US))

SWT2_CPB: 

• We have been working with campus networking. Our internal and external networking has been upgraded with various fiber cables step-by-step. The results are that we now have an external bandwidth of 80 Gbps. 60 Gbps is dedicated through LHCONE. Additional 20 Gbps is commodity. 

• LEARN performed a network upgrade that helped  increase our bandwidth through ESNET to 60 Gbps on 9/4, as part of this upgrade.

• Due to security concerns, we communicated with OSG and the perfSONAR team for assistance with our perfSONAR nodes and temporarily powered them off for now. 

• They provided us helpful information so that we can test a new configuration and are testing this. 

• We discovered tests that showed “UNKNOWN” and “CRITICAL” status on the site availability and reliability dashboard on 9/2 and 9/3 when our site was working perfectly well. 

• We are communicating with the appropriate people to investigate and fix this. 

• We see fast fluctuation of slots of running jobs. We are concerned about how efficiently our job slots are being used. Instead of the plot being flat across, they are fluctuating heavily. We are looking into this and communicating with the appropriate people to investigate. 

• GGUS-Ticket-ID: #1003053 - IGTF CRLs & fetch-crl SHA1 signature validation

• We are continuing to investigate this and are planning to implement a fifth XRootD Proxy server. 

• We will see if the fifth XRootD Proxy experiences this issue, while also continuing to inspect the one XRootD Proxy server that this error is involved with.

OU:

• Storage migration still underway. Most OU_OSCER_ATLAS_TEST HC jobs are succeeding, but not all of them. Please see JIRA ticket ATLDDMOPS-5854 for more details.
• Sorry, can't make today becaus of the XRootD workshop.