US ATLAS Tier 2 Technical

Wednesday 28 Oct 2026, 11:00 → 12:00 US/Eastern

Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), Shawn Mc Kee (University of Michigan (US))

Meeting to discuss technical issues at the US ATLAS Tier 2 site. The primary audience is the US Tier 2 site administrators but anyone interested is welcome to attend.

11:00 → 11:10 Introduction

Speakers: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

11:10 → 11:20 TW-FTT

Speakers: Eric Yen, Felix.hung-te Lee (Academia Sinica (TW)), Yi-Ru Chen (Academia Sinica (TW))

• Site is running smoothly.
• ggus:1003998 TW-FTT: Storage Token Support: Deadline - 01.11.2026
  • We have applied the token configuration and enabled ZTN authentication. WebDAV is now working, but XRootD is still failing. Test Link

  • Issue: The same SciToken subject is mapped to different identities depending on the operation. EOS experts recommended upgrading EOS version.

  • We’re planning the upgrade.

11:20 → 11:30 AGLT2

Speakers: Daniel Hayden (Michigan State University (US)), Philippe Laurens (Michigan State University (US)), Shawn Mc Kee (University of Michigan (US)), Dr Wendy Wu (University of Michigan)

11:30 → 11:40 MWT2

Speakers: Aidan Rosberg (Indiana University (US)), David Jordan (University of Chicago (US)), Farnaz Golnaraghi (University of Chicago (US)), Fengping Hu (University of Chicago (US)), Fred Luehring (Indiana University (US)), Judith Lorraine Stephen (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

• MWT2 set offline for ~2hrs Oct 27
  • Due to Varnish on the UChicago AF going down
  • Many hosts didn't failover properly (to investigate)
• GGUS 1004286 MWT2 xcache.uc.mwt2.org in RED
  • Known. We had to firewall off the analysis facility over the weekend. Should be back up today
• UChicago network hardware quotes in - still waiting on optics quote

11:40 → 11:50 NET2

Speakers: Eduardo Bach (University of Massachusetts (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), William Axel Leight (University of Massachusetts Amherst)

Downtime from the 16-18th due to expired certificates, as our certificate provider had suspended the issuance of IGTF certificates.  After a couple of days the suspension (in place for weeks at that point) was finally lifted and we were able to get new certificates.  There was a further issue with the new certificates, but that was on the Harvester side and was resolved by Fernando.

Last weekend implemented a configuration change that should eliminate backoff errors on the non-dense nodes, but accidentally introduced a bug, leading to a number of nodes being dropped from the cluster.  This was fixed and we are back to normal running.

Last dense servers were added back to the cluster, we are back to 14.5k slots.

Suspended tape writing for the ongoing tape recall campaign.

11:50 → 12:00 SWT2

Speakers: Andrey Zarochentsev (University of Texas at Arlington (US)), Horst Severini (University of Oklahoma (US)), Kaushik De (University of Texas at Arlington (US)), Mark Sosebee (University of Texas at Arlington (US)), Zachary Thomas Booth (University of Texas at Arlington (US))

OU:

• Site running well, including new storage
• Old storage has been officially decommissioned
• Some nodes had problems accessing storage, those have been fixed and additional monitoring is in place to prevent black hole nodes
• Sorry, can't make it today