US ATLAS Tier 2 Technical

Wednesday 5 Aug 2026, 11:00 → 12:00 US/Eastern

Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), Shawn Mc Kee (University of Michigan (US))

Meeting to discuss technical issues at the US ATLAS Tier 2 site. The primary audience is the US Tier 2 site administrators but anyone interested is welcome to attend.

11:00 → 11:10 Introduction

Speakers: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

• Good running over the past two weeks.
  • Some job failures at NET2 due to the back off errors.
  • AGLT2 drained on 7/28 due to an unplanned side effect of mitigating a security issue.
  • CPB had reduced slots due to power work and to avoid an XRootD memory issue. All job slots returned to service on 8/1. 
  • CPB received no work for about 12 hours on 7/28
  • OU drained on 7/28 due to failed dark data reporting.
  • TW-FTT some job failures but generally looking good.
• I have no news about when the funding will become available though I believe it will be soon.
  • I asked the bosses for an update but have not heard back.
  • There are discussions about possibly extending the current CA by a month or two which would ease the deadline for ordering with the end of CA funds.

11:10 → 11:20 TW-FTT

Speakers: Eric Yen, Felix.hung-te Lee (Academia Sinica (TW)), Yi-Ru Chen (Academia Sinica (TW))

• Our outbound bandwidth was saturated during the past two weeks (7/23–7/26).

11:20 → 11:30 AGLT2

Speakers: Daniel Hayden (Michigan State University (US)), Philippe Laurens (Michigan State University (US)), Shawn Mc Kee (University of Michigan (US)), Dr Wendy Wu (University of Michigan)

Long term: UM server room needs refresh of power and UPS infrastrucutre
    while UM is also considering a new data center on campus.
Short term: The R760xd2 we would like to purchase don't fit in the racks.
Mid term solution: replace 2 racks with deeper/wider racks
    One LSA rack was replaced and added to AGLT2
    The other was an existing AGLT2 rack, replaced July 24
    No site downtime. 
    11 worker nodes were drained.
    Moved 8 dcache servers, one at a time.
    Those pools have been rdonly for over 24h. 
    MSU had a cached copy of the “hot files.”
    Relocated the 8 storage servers in 2 hours
    Small number of jobs (~110 jobs) failed due to data access failure.

 Blacklisted due to Worker Node configuration problem, for 17h, July 27-28
    Ansible policy to mitigate threats (preventing user namespace) had been mistakenly extended to the WNs.

11:30 → 11:40 MWT2

Speakers: Aidan Rosberg (Indiana University (US)), David Jordan (University of Chicago (US)), Farnaz Golnaraghi (University of Chicago (US)), Fengping Hu (University of Chicago (US)), Fred Luehring (Indiana University (US)), Judith Lorraine Stephen (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

• IU had PDU upgrades and maintenance across several racks last week
• Upgraded Elasticsearch from 9.2.0 to 9.4.4
• GGUS Ticket #1003446 for mwt2 XCache, it was a brief monitoring failure traced to an upstream BNL origin-file issue. No MWT2 site intervention was needed
• Planned CE drains for security updates caused an “unknown” monitoring status. Requesting A/R recalculation

11:40 → 11:50 NET2

Speakers: Eduardo Bach (University of Massachusetts (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), William Axel Leight (University of Massachusetts Amherst)

Tape remains in downtime. During a brief discussion with the DDM team at S&C Week, we confirmed that the extended downtime required to implement the new internal dCache tape workflow can be scheduled at any time, since secondary copies of the data are available elsewhere. The hardware issue that triggered the current downtime has allowed us to begin part of this work while giving IBM sufficient time to investigate the performance problem thoroughly. The current plan is to restore tape services on Friday.

Efficiency has been intermittently low, occasionally very low, owing to persistent backoff errors in the dense servers.  We are in touch with Dell to try to figure out what is going on.  While this goes on, we will remove the dense servers from the cluster, as soon as we can arrange for alternative storage for the various necessary services running on them.

11:50 → 12:00 SWT2

Speakers: Andrey Zarochentsev (University of Texas at Arlington (US)), Horst Severini (University of Oklahoma (US)), Kaushik De (University of Texas at Arlington (US)), Mark Sosebee (University of Texas at Arlington (US)), Zachary Thomas Booth (University of Texas at Arlington (US))

SWT2_CPB: 

• Campus Facilities Management performed electrical work and transitioned the data center from building to generator power from 7/24-7/25. 

• We took time to prepare for this by reducing our power usage (by partially draining and powering off our oldest worker nodes, plus other nodes that would not affect the production cluster), providing additional battery runtime on our UPS and reducing temperature in the room. 

• There were no issues. We never lost power, the transition on and off the generation went smoothly, and we returned all worker nodes to their previous state on 7/31. 

• We set the SWT2_CPB_K8S offline, drained K8S worker nodes, and plan on rebuilding these nodes to be used in the SWT2_CPB PQ. 

• There may have been a potential issue in the XRootD Redirector configuration that may have caused errors in the SWT2_CPB_K8S PQ. The change was made during implementing a second redirector, but we are unsure. Jobs started to succeed right before draining after reverting this change. 

• Experienced memory leak issue with all four of our DTNs.

• Due to 330k certificate CRLs being revoked by HARICA CA, causing memory/space issues on GRID middleware. 

• We implemented jemalloc, which significantly improved memory issues on all of our DTNs. We have not had any XRootD Proxy service fail so far after this change. 

• We also modified the service to automatically restart (with limits) and to have OOM stop the processes more cleanly. 

• Troubleshooting and diagnosing this issue last week, we contacted XRootD experts, tested XRootD version 5.9.6 in our test cluster, added additional memory, and tried other methods for minimizing or solving issues at our site. 

• Implemented additional monitoring and alerts of our XRootD Proxy servers. 

• GGUS-Ticket-ID: #1003053 - IGTF CRLs & fetch-crl SHA1 signature validation (SWT2_CPB)

• Removed old certs, but did not work. Performing transfer tests after change failed. We reverted this change. 

OU:

• se0's checksum issue should be solved, so should be able to start migrating storage now
• Held/canceled jobs issue disappeared again, still no idea what that was