US ATLAS Tier 2 Technical

Wednesday 28 Oct 2026, 11:00 → 12:00 US/Eastern

Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), Shawn Mc Kee (University of Michigan (US))

Meeting to discuss technical issues at the US ATLAS Tier 2 site. The primary audience is the US Tier 2 site administrators but anyone interested is welcome to attend.

11:00 → 11:10 Introduction

Speakers: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

Quick links:

• Capacity spreadsheet
• Services spreadsheet 

Upcoming meetings:

• WLCG/HSF Workshop 2026 [Nov 2nd -- 6th, Bologna]
• SC26 [Nov 16nd -- 21st, Chicago]

Open tickets and issues:

• ggus 1004286  MWT2 XCache server in red
• ggus 1003998 TW-FTT: Storage Token Support: Deadline - 01.11.2026
  
  
• NET2_MCTAPE blacklisted in DDM (DISKSPACE). More info

Equipment Purchasing with End of CA Funds

• Got quotes back from Dell for about $5 million of equipment.
  • The pricing is not good.
  • The network switches are especially costly because Dell is now licensing the firmware (Sonic) for in excess of $1000/year.
• The managers will discuss the equipment purchases at today's Facility Management meeting.

Tier 2 Site Performance

• Generally quite good.
• All sites drained on October 17-18 due to Panda system token failure.
• MWT2 had most jobs fail on Oct 22nd when OSG unintentionally changed something in the OASIS repository causing some root certificates to be unavailable.

11:10 → 11:20 TW-FTT

Speakers: Eric Yen, Felix.hung-te Lee (Academia Sinica (TW)), Yi-Ru Chen (Academia Sinica (TW))

• Site is running smoothly.
• ggus:1003998 TW-FTT: Storage Token Support: Deadline - 01.11.2026
  • We have applied the token configuration and enabled ZTN authentication. WebDAV is now working, but XRootD is still failing. Test Link

  • Issue: The same SciToken subject is mapped to different identities depending on the operation. EOS experts recommended upgrading EOS version.

  • We’re planning the upgrade.

11:20 → 11:30 AGLT2

Speakers: Daniel Hayden (Michigan State University (US)), Philippe Laurens (Michigan State University (US)), Shawn Mc Kee (University of Michigan (US)), Dr Wendy Wu (University of Michigan)

Getting switch quotes from Dell for 7year support and for updating our existing and recent switches to SONiC.
    But will likely not update existing switches.  
    Instead plan on exploring the community version of SONiC when OS10 licenses run out.

Planning on updating dCache to 11.2.10 today
   " This release fixes the compatibility with recently released xrootd clients.
    Sitesare advised to update their xrootd doors as soon as possible to ensure user operation.”

did PHP updates on the webservers to address the security issues

had 2 GGUS tickets, all closed

• GGUS #1004157 — Pilot failures at AGLT2
  Opened Oct 14. Thousands of PanDA pilot failures were traced to worker node c-16-12.aglt2.org. The failures were due to CVMFS being unable to serve the pilot tarball (pilot3-3.14.4.19.tar.gz), causing all pilots landing on that node to fail immediately at startup with exit code 64. Diagnosed the CVMFS issue on c-16-12.aglt2.org, fixed it, and returned the node to production the same day. Status: SOLVED Oct 14.

• • GGUS #1004200 — AGLT2-S3_LOCALGROUPDISK outage, Oct 16–17 — CLOSED

  11:40 UTC Oct 16, OSiRIS OSD #501 (um-stor06) hung — the daemon stopped responding but the disk was physically healthy (SMART OK, no I/O errors). A placement group (PG 43.f8d) got stuck peering for ~25 hours, causing 256 slow ops to pile up. RGW requests touching that PG blocked forever, eventually exhausting the request thread pool on all three port-80 RGWs (UM, MSU, wsu-xfer01). With all RGW accept queues full, the xrootd S3 gateway got 95 connections stuck in SYN-SENT and returned HTTP 500 to callers. ~8K transfer failures were reported to ATLAS shifters. xrootd itself was healthy throughout — the problem was entirely in the OSiRIS Ceph backend.

  Fix (Oct 17 ~13:00 UTC): ceph osd down 501, then restarted osd.501. The stuck PG peered immediately, slow ops cleared, and all three RGWs recovered on their own without a restart. Transfers back to normal; ticket closed by Armen Vartapetian Oct 19.

11:30 → 11:40 MWT2

Speakers: Aidan Rosberg (Indiana University (US)), David Jordan (University of Chicago (US)), Farnaz Golnaraghi (University of Chicago (US)), Fengping Hu (University of Chicago (US)), Fred Luehring (Indiana University (US)), Judith Lorraine Stephen (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

• MWT2 set offline for ~2hrs Oct 27
  • Due to Varnish on the UChicago AF going down
  • Many hosts didn't failover properly (to investigate)
• GGUS 1004286 MWT2 xcache.uc.mwt2.org in RED
  • Known. We had to firewall off the analysis facility over the weekend. Should be back up today
• UChicago network hardware quotes in - still waiting on optics quote

11:40 → 11:50 NET2

Speakers: Eduardo Bach (University of Massachusetts (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), William Axel Leight (University of Massachusetts Amherst)

Downtime from the 16-18th due to expired certificates, as our certificate provider had suspended the issuance of IGTF certificates.  After a couple of days the suspension (in place for weeks at that point) was finally lifted and we were able to get new certificates.  There was a further issue with the new certificates, but that was on the Harvester side and was resolved by Fernando.

Last weekend implemented a configuration change that should eliminate backoff errors on the non-dense nodes, but accidentally introduced a bug, leading to a number of nodes being dropped from the cluster.  This was fixed and we are back to normal running.

Last dense servers were added back to the cluster, we are back to 14.5k slots.

Suspended tape writing for the ongoing tape recall campaign.

11:50 → 12:00 SWT2

Speakers: Andrey Zarochentsev (University of Texas at Arlington (US)), Horst Severini (University of Oklahoma (US)), Kaushik De (University of Texas at Arlington (US)), Mark Sosebee (University of Texas at Arlington (US)), Zachary Thomas Booth (University of Texas at Arlington (US))

SWT2_CPB: 

• We added a fifth XRootD proxy (gk09) server back into our DNS round robin on 10/14. We monitored and did not see any major issues with transfers with this change. 

• We issued a new certificate for this XRootD proxy server beforehand with a new CA CN attribute InCommon RSA IGTF Server CA 4. We see that CRLs are delayed in updating this certificate. We are communicating with others to investigate this issue specific to this certificate. 

• We removed the problematic XRootD proxy server (gk05) related to the GGUS-Ticket-ID: #1003053. 

• After removal of this server and the addition of gk09, we stopped seeing these errors with the BEIJING-LCG site. The ticket has been marked resolved. 

• We migrated data from one of our storage servers and noticed that the method we used to check that all deleted files during this time is accounted for no longer works. Deletions are not found in the XRootD Proxy log files. We believe this may be some central change causing this, but are still investigating. 

• We now use a different method where we run a Python script to use an API to query CERN grafana monitoring. It requires a token, but we are limited to querying ten thousand deleted files at a time and there appears to be no way around this as far as we have tested. 

OU:

• Site running well, including new storage
• Old storage has been officially decommissioned
• Some nodes had problems accessing storage, those have been fixed and additional monitoring is in place to prevent black hole node