US ATLAS Tier 2 Technical

Wednesday 30 Sept 2026, 11:00 → 12:00 US/Eastern

Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), Shawn Mc Kee (University of Michigan (US))

Meeting to discuss technical issues at the US ATLAS Tier 2 site. The primary audience is the US Tier 2 site administrators but anyone interested is welcome to attend.

11:00 → 11:10 Introduction

Speakers: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

Quick links:

• Capacity spreadsheet
• Services spreadsheet 

General news

• Still waiting for sites to confirm if they will purchase with Dell.
• Dell delivery is expected in February.

Upcoming meetings:

• 57th LHCOPN-LHCONE meeting [Oct 1st -- 2nd, Naples]
• #85 ATLAS S&C week  [Oct 5th -- 9th, CERN]
• HEPiX Fall 2026 Workshop [Oct 19th -- 23rd, Nebraska] -- Including USATLAS/USCMS sessions. At least one person per site should try to attend.
• WLCG/HSF Workshop 2026 [Nov 2nd -- 6th, Bologna]
• SC26 [Nov 16nd -- 21st, Chicago]

Open tickets:

• ggus:1003998 TW-FTT: Storage Token Support: Deadline - 01.11.2026

11:10 → 11:20 TW-FTT

Speakers: Eric Yen, Felix.hung-te Lee (Academia Sinica (TW)), Yi-Ru Chen (Academia Sinica (TW))

• After the site recovered from the downtime on Sep. 22, all services are back to normal.

11:20 → 11:30 AGLT2

Speakers: Daniel Hayden (Michigan State University (US)), Philippe Laurens (Michigan State University (US)), Shawn Mc Kee (University of Michigan (US)), Dr Wendy Wu (University of Michigan)

smooth operation

drained and update work nodes to HTCondor 25.0.14 to address the security issues, also updated firmware and kernel (MSU only) and rebooted the work nodes

MSU updated firmware and rebooted for all the storage nodes as well

ggus ticket for S3 storage service outage (resolved, it was caused by the OSiRIS Ceph storage at MSU losing a Rack switch)

Deployed a second Amanda backup server with retired Storage (MD3060e) at UM site as the tape library server is out of warranty and has a bad tape driver

 

11:30 → 11:40 MWT2

Speakers: Aidan Rosberg (Indiana University (US)), David Jordan (University of Chicago (US)), Farnaz Golnaraghi (University of Chicago (US)), Fengping Hu (University of Chicago (US)), Fred Luehring (Indiana University (US)), Judith Lorraine Stephen (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

• Did a rolling update of Condor to 25.0.14 (From 09/17 to 09/19)
• Still working on procurement
• Network routes fixed from UC to IU and UIUC

11:40 → 11:50 NET2

Speakers: Eduardo Bach (University of Massachusetts (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), William Axel Leight (University of Massachusetts Amherst)

Smooth running except for one black hole node on Monday.

11:50 → 12:00 SWT2

Speakers: Andrey Zarochentsev (University of Texas at Arlington (US)), Horst Severini (University of Oklahoma (US)), Kaushik De (University of Texas at Arlington (US)), Mark Sosebee (University of Texas at Arlington (US)), Zachary Thomas Booth (University of Texas at Arlington (US))

SWT2_CPB: 

• We asked to have tests for storage and hammercloud jobs run in our test cluster, so that we can test changes before implementing them into the production cluster. 

• We have decided against changing the type of SE and turning on HammerCloud jobs for this cluster.

• We will perfom tests ourselves to verify jobs run without issues and that storage is functioning properly when testing changes in this cluster. 

• We experienced issues with cooling in the data center on 9/21 and drained our oldest worker nodes to alleviate these issues until proper cooling was restored. We put these nodes back into service at 5:00 p.m. that same day. 

• GGUS-Ticket-ID: #1003053 - IGTF CRLs & fetch-crl SHA1 signature validation

• We configured a fifth XRootD Proxy server (gk09) and are waiting for campus networking to add this into our DNS round robin. 

• We performed tests and saw that gk09 does not have any issues communicating with the BEIJING site, while the issue still persists with gk05. 

OU:

• Running well, except got set offline because of 'copytool' change made yesterday
• Should be ready to migrate to new storage, except for this copytool issue
• OU_OSCER_ATLAS_TEST HC jobs ran successfully, so we need to figure out how to match the copytool settings in other PQs, and then hopefully everything will work across all PQs
• There seem to be a high failure rate on c272-4. I'm not sure there's an actual problem with those nodes, or is jobs just happen to end up on these nodes and failing because of inconsistent copytool settings. But I'll ask the admins to drain these nodes temporarily, to see whether that helps
• Also, there seem to be two OU_OSCER_GPU PQs now, one in production and one in analysis:
• https://bigpanda.cern.ch/dash/region/?region=US&jobtype=prod&splitby=jobtype&display_limit=100
• https://bigpanda.cern.ch/dash/region/?jobtype=analy&splitby=jobtype&region=US&display_limit=100
• The second one is the correct one. Where's the first one coming from?
• New OU_OSCER_LOCALGROUPDISK overfull. Should we increase the size?