US ATLAS Tier 2 Technical

Wednesday 11 Nov 2026, 11:00 → 12:00 US/Eastern

Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), Shawn Mc Kee (University of Michigan (US))

Meeting to discuss technical issues at the US ATLAS Tier 2 site. The primary audience is the US Tier 2 site administrators but anyone interested is welcome to attend.

11:00 → 11:10 Introduction

Speakers: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

Quick links:

• Capacity spreadsheet
• Services spreadsheet 

Upcoming meetings:

• SC26 [Nov 16nd -- 21st, Chicago]

Open tickets and issues:

• ggus 1003998 TW-FTT: Storage Token Support: Deadline - 01.11.2026

Recent production quality

• The sites have been running well.
  • MWT2_IU maintenance lowered production
  • MWT2 drained today due to another issue reading certificate updates from an OSG cvmfs repository.
  • NET2 drained partly because of dCache upgrade on Tuesday

Procurement:

• All sites should have received authorization for the end of CA purchases.
  • This increment covers October-January.
  • The February-March increment paperwork is expected in mid-December.
  • NCE after that (probably for 1 year).
• Dell quotes expire November 20.
  • Pricing could go up after November 20.

11:10 → 11:20 TW-FTT

Speakers: Eric Yen, Felix.hung-te Lee (Academia Sinica (TW)), Yi-Ru Chen (Academia Sinica (TW))

• Site is running smoothly.
• ggus:1003998 TW-FTT: Storage Token Support: Deadline - 01.11.2026
  • Our storage admin is still working on the upgrade.

11:20 → 11:30 AGLT2

Speakers: Daniel Hayden (Michigan State University (US)), Philippe Laurens (Michigan State University (US)), Shawn Mc Kee (University of Michigan (US)), Dr Wendy Wu (University of Michigan)

Overall good running.  New ticket today, to investigate.
[GGUS-Ticket-ID: #1004423] timeout on webdav with data deletion (80% efficiency )

Started rolling rebuild of the dCache pool nodes from RHEL 9 to RHEL 10
Hoping to have about half the dcache pool on EL10 for the December data challenge,  for comparison

Sporadic cvmfs issues on work nodes which sometime recover by themselves,
but can sometime crash the work nodes or corrupt it to become a black hole.

Efforts to strengthen site security and threat monitoring (MISP)

MSU had difficulties replacing the internal satellite-capsule certificates.
Was preventing access to repos, rebuilding nodes, etc.
Believed to be almost resolved.

MSU cannot currently obtain IGTF certs.
MSU IT is aware and looking for a solution.
Not a global site problem because UM can get the certs.

11:30 → 11:40 MWT2

Speakers: Aidan Rosberg (Indiana University (US)), David Jordan (University of Chicago (US)), Fred Luehring (Indiana University (US)), Judith Lorraine Stephen (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

• Working out remaining purchases we'd like to make and getting quotes for specialized machine configs
• IU compute was offline Monday (Nov 9) for PDU upgrades
• UIUC compute offline Friday (Nov 6) for security updates
• Queue drained this morning due to our CA cert sanity check using a now deprecated CA file
  • Updated check and now filling back up
• xcache back in prod from AF site emergency a couple weeks ago

11:40 → 11:50 NET2

Speakers: Eduardo Bach (University of Massachusetts (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), William Axel Leight (University of Massachusetts Amherst)

Had an issue with storage dumps not being updated, that turned out to be due to a certificate mismatch stemming from the issue with renewing IGTF certificates.

Blacklisted briefly on the 4th after UMass IT accidentally cut the cluster off from the internet while trying to fix a security issue.

dCache upgrade on last Monday

• Upgraded dCache from 9.2.47 to 11.2.7
• Upgraded the ENDIT tape interface to alpha.20. 
• Enabled SciTags/Firefly reporting on pools. Verified outgoing start/end messages, including transfer and storage statistics. 
• Enabled per-domain JVM monitoring: Java heap and memory-pool usage, garbage collection, CPU, threads and file descriptors. Will is building OpenSearch/Grafana plots. 
• Raised the head-node WebDAV request limits from (default) 1,000 to 5,000 requests/second per door, and 250 to 1,250 per client, based on historical traffic. 
• Centralized Kafka billing through the head-node Billing service, with JSON events and local file billing retained. 

Remaining work:  address intermittent frontend monitoring throttling, upgrade to the latest minor version.

Java remains on version 17 and PostgreSQL on 13.23. 

11:50 → 12:00 SWT2

Speakers: Andrey Zarochentsev (University of Texas at Arlington (US)), Horst Severini (University of Oklahoma (US)), Kaushik De (University of Texas at Arlington (US)), Mark Sosebee (University of Texas at Arlington (US)), Zachary Thomas Booth (University of Texas at Arlington (US))

SWT2_CPB: 

• One PowerEdge R740xd2 storage server’s PERC experienced issues, causing unexpected and frequent restarts on 11/5. 

• We declared a downtime from CRIC (Unsure if this took effect, may need to do it a different way) that same evening and worked with the vendor to thoroughly check the server and replace the PERC. 

• We stopped seeing issues with this server and ended our downtime on 11/6. 

• We saw A/R errors two weeks ago. We saw production tasks did not show more errors. We observed some issues with CVMFS access to our Frontier Squid servers.

• We built a new EL9 Frontier Squid server and are currently testing this in our test cluster. Once finished, we will implement a new server with better hardware than our current Frontier Squids in the production cluster. 

OU:

• Running well. getting good opportunistic throughput
• Opportunistic GPU throughput varies greatly, depending on local OU demand
• Have made some changes to GPU PQs: Turned off ANALY_OU_OSCER_GPU_TEST PQ for now, since that just duplicates HC jobs from OU_OSCER_GPU_ PQ
• Changed local SLURM partition for all ATLAS GPU jobs from sooner_gpu_test to gpu, since that partition contains all public GPU nodes
• se0 auto-updated to XRootD 5.9.8 this morning, seems to have gone well