US ATLAS Tier 2 Technical

Wednesday 14 Oct 2026, 11:00 → 12:00 US/Eastern

Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), Shawn Mc Kee (University of Michigan (US))

Meeting to discuss technical issues at the US ATLAS Tier 2 site. The primary audience is the US Tier 2 site administrators but anyone interested is welcome to attend.

11:00 → 11:10 Introduction

Speakers: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

Quick links:

• Capacity spreadsheet
• Services spreadsheet 

Upcoming meetings:

• HEPiX Fall 2026 Workshop [Oct 19th -- 23rd, Nebraska] -- Including USATLAS/USCMS sessions. At least one person per site should try to attend.
• WLCG/HSF Workshop 2026 [Nov 2nd -- 6th, Bologna]
• SC26 [Nov 16nd -- 21st, Chicago]

Open tickets and issues:

• ggus 1004157 AGLT2: pilot failures
• ggus 1004150 TW-FTT: transfers fail with "Connection terminated abruptly; Status of TPC request unknown"
• ggus 1003998 TW-FTT: Storage Token Support: Deadline - 01.11.2026
  
  
• NET2_MCTAPE blacklisted in DDM (DISKSPACE). More info
• OU_OSCER_ATLAS_SCRATCHDISK blacklisted in DDM (manual). More info

11:10 → 11:20 TW-FTT

Speakers: Eric Yen, Felix.hung-te Lee (Academia Sinica (TW)), Yi-Ru Chen (Academia Sinica (TW))

• ggus:1003998 TW-FTT: Storage Token Support: Deadline - 01.11.2026
  • We applied the token configuration. WebDAV is now working successfully, but XRootD is still failing. Link
• ggus:1004150  TW-FTT:transfers fail with "Connection terminated abruptly; Status of TPC request unknown" 

  • Yesterday, the disk servers experienced an unexpected system error, which caused a service outage. All servers have now recovered.

11:20 → 11:30 AGLT2

Speakers: Daniel Hayden (Michigan State University (US)), Philippe Laurens (Michigan State University (US)), Shawn Mc Kee (University of Michigan (US)), Dr Wendy Wu (University of Michigan)

overall smooth running

In the  past a few days, a few work nodes had cvmfs issues and became blackhole nodes, we had to drain the nodes and reinstall cvmfs (empty the cache directory) to fix cvmfs (ggus ticket 1004157 is resolved by fixing cvmfs on the work node)

The s3 storage gateway caused 2 tickets: 1) ceph failures 2) xrootd server failure, Shawn fixed it, mainly used by leaking memory and state that could only be cleared with restarting the xrootd service

11:30 → 11:40 MWT2

Speakers: Aidan Rosberg (Indiana University (US)), David Jordan (University of Chicago (US)), Farnaz Golnaraghi (University of Chicago (US)), Fengping Hu (University of Chicago (US)), Fred Luehring (Indiana University (US)), Judith Lorraine Stephen (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

• Knocked offline briefly on Oct 12th due to a UChicago upstream DNS issue - fixed same day
• IU and UC working with respective networking teams to plan network hardware refreshes
• Fixed asymmetric routing between UC and UIUC and UC and IU

11:40 → 11:50 NET2

Speakers: Eduardo Bach (University of Massachusetts (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), William Axel Leight (University of Massachusetts Amherst)

Tape Storage: Scheduled maintenance was carried out on the tape library's mechanical system, with short downtimes planned for 1 and 5 October, each expected to last approximately 1–2 hours. The maintenance on 5 October was completed successfully. An additional brief tape downtime occurred on 7 October. There is an ongoing ATLAS tape recall campaign running. There was a temporary race for space for flush and restore requests, solved.

OKD Cluster: Two previously unavailable high-density compute nodes were successfully returned to the cluster following disk replacements. The replacement disks introduced controller-related complications with the software RAID configuration.We implemented a workaround, allowing both nodes to be restored to service.

11:50 → 12:00 SWT2

Speakers: Andrey Zarochentsev (University of Texas at Arlington (US)), Horst Severini (University of Oklahoma (US)), Kaushik De (University of Texas at Arlington (US)), Mark Sosebee (University of Texas at Arlington (US)), Zachary Thomas Booth (University of Texas at Arlington (US))

SWT2_CPB: 

• We upgraded our UPS with new battery modules, which will be good for another five years. 

• We completed the configuration and performed some testing of our fifth XRootD Proxy server. It was added to our DNS round robin on 10/1. There were transfer issues, and we communicated with campus networking to have it removed from DNS. 

  • We discovered an issue with the certificate. 

  • We worked with campus networking to have the certificate fixed and received a new certificate. 

• Our perfSONAR machines remain offline for now. 

  • Continuing to test new perfSONAR configuration instructions and scripts. We are continuing to work with OSG to help improve configuration instructions.. 

• We drained our oldest compute nodes, R410s, on 10/8 due to a cooling issue in the data center. Facilities Management is waiting for delivery of a replacement backup pump in November.  We plan to keep these off for now to reduce temperature in the room till this pump is replaced.

• GGUS-Ticket-ID: #1003053 - IGTF CRLs & fetch-crl SHA1 signature validation

  • Will test gk09, our fifth XRootD Proxy server, to see if it has this same issue and potentially replace the problematic DTN with gk09.

OU:

• Site running well
• Storage migration finally finished, everything looks good
• Still need to retire old storage in rucio, will happen in a few weeks