US ATLAS Tier 2 Technical

Wednesday 27 May 2026, 11:00 → 12:00 US/Eastern

Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), Shawn Mc Kee (University of Michigan (US))

Meeting to discuss technical issues at the US ATLAS Tier 2 site. The primary audience is the US Tier 2 site administrators but anyone interested is welcome to attend.

11:00 → 11:10 Introduction

Speakers: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

• Rafael is unable to attend.
• Excellent production over the last two weeks.
  • There were almost no disruptions to production.
  • Inbound transfers was 95%-98% successful depending on site.
  • Outbound transfers was 97%-99% successful depending on site.
  • Wall time weighted job success efficiency was 93%-99% depending on site.
• The 2027 pledge recommendations were set at the March C-RSG meeting.
  • As expected there was no increase in the CPU pledge and about an 8% increase in the storage pledge.
• The procurement plans for the FY25, FY26, and end of CA equipment funding are due next Monday.
  • We will have a special meet next week (probably on Friday morning) to compare notes and solicit feedback from management.
  • I will put the MWT2 plan out later today as an example.
    • Please just hit the high points and concisely describe how you expect to spend the money.
• And as has been the case, I still have no information on when the funding will be released.
  • The funding will be $1.7 million per site.

11:10 → 11:20 TW-FTT

Speakers: Eric Yen, Felix.hung-te Lee (Academia Sinica (TW)), Yi-Ru Chen (Academia Sinica (TW))

• We had a switch H/W issue on May 14 at 21:00 (UTC), and it recovered on May 15 at 02:00 (UTC).
• A network provider circuit interruption (unplanned maintenance) occurred on 5/17. We declared an unscheduled downtime, and the service recovered on 5/19.

11:20 → 11:30 AGLT2

Speakers: Daniel Hayden (Michigan State University (US)), Philippe Laurens (Michigan State University (US)), Shawn Mc Kee (University of Michigan (US)), Dr Wendy Wu (University of Michigan)

  14-May: Cleaned out all the dark data (186TB) from atlasdatadisk (and all space tokens)
    2 steps were required
    Both were aimed at clearing the (hopefully) last remaining effects from our Dec-2024 dcache database problem.
    Find all files set as sticky in the dcache pools, but without records in the chimera db (t_inodes table)
      Set these files sticky=off and run sweeper purge to let the pools remove them.
      However we also had to update the SpaceManager (purge file pnfsid) for these files
      otherwise the spacetoken was still counting their space as used.
    We had previously cleaned up some of our pools, after being drained, by setting sticky=off on the remaining files
      All MSU pools during the EL7 to EL9 upgrade + Some UM pools to correct the RAID stripe size
      But we had not updated the SpaceManager at that time.
      So we compared the set of pnfsids in the spacemgr db (srmspacefile table) and the chimera db (t_inodes table)
      to find all files which exist in spacemgr but not in chimera
      and purged them from the SpaceManage cell (using “purge file pnfsid”)

11:30 → 11:40 MWT2

Speakers: Aidan Rosberg (Indiana University (US)), David Jordan (University of Chicago (US)), Farnaz Golnaraghi (University of Chicago (US)), Fengping Hu (University of Chicago (US)), Fred Luehring (Indiana University (US)), Judith Lorraine Stephen (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))

• Patched the ssh-keysign vulnerability
• Set offline on the 19th because BNL stratum 1 was down
• Set offline on the 26th due to high number of transfers from user jobs
• Fred is working on benchmarking 
• Updated the OSG security list

11:40 → 11:50 NET2

Speakers: Eduardo Bach (University of Massachusetts (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), William Axel Leight (University of Massachusetts Amherst)

Thanks to Ivan for noticing a problem with the accounting for March, this was fixed.

Second data challenge with Prague showed that the previous 100 Gb/s limit from ESNet is now gone.  The test was able to reach the nominal value of 400 Gb/s, but more parallel transfers were required for this than expected.  This is probably due to the distance, though more investigation will be required to confirm this.  There was a brief blacklisting during this test, likely because of the high number of parallel transfers required.

Opened a ticket with OSG last week to change the admin and security contacts to be lists rather than people, which they seemed reluctant to do.  This still hasn't happened, we'll ping the ticket again.

Last week the tape recall workflow stalled because the tape-facing dCache pool reached 100% usage with cached/removable files. From dCache’s point of view, this space could be reclaimed, but the tape system only sees the filesystem as full and therefore stopped recalling additional files. This failure mode was previously masked because recalls and transfers happened as one automatic movement: files were staged from tape and then promptly transferred onward.

The new WLCG workflow separates the process into a bring-online request followed later by the actual copy transfer. Because the second step competes in FTS with normal outbound transfers from the main dCache cluster, recalled files now remain on the tape-facing pool much longer, allowing cached files to accumulate and eventually block the tape system. We are redesigning the internal workflow so that can be avoided.

11:50 → 12:00 SWT2

Speakers: Andrey Zarochentsev (University of Texas at Arlington (US)), Horst Severini (University of Oklahoma (US)), Kaushik De (University of Texas at Arlington (US)), Mark Sosebee (University of Texas at Arlington (US)), Zachary Thomas Booth (University of Texas at Arlington (US))

SWT2_CPB: 

• Migration of our last EL7 storage is complete. No EL7 storage is being used to host data. 

• Checks are being done to ensure migration of data looks fine.

• We will rebuild our last ME4084 to EL9 once checks are complete.  

• We rebuilt the previous ME4084 that finished migrating and performing checks. 

• Applied the most recent security fix for a new vulnerability due to a kernel bug and specific suid binaries to the production cluster.

• We will be rebuilding Slate01 to repurpose it as our internal monitoring server. This will be an upgrade that improves the performance of our internal monitoring. 

• We experienced a brief power issue at the chilled water plant which caused the temperature to start rising on Sunday (5/24) around 8:00 a.m. 

• Contacted the appropriate people to check this and received status updates. 

• They added an additional chiller, which reduced our temperature.

• We still see a very slight increase in temperature on the following Tuesday (5/26). We are looking into this. 

• We experienced a brief power outage on Monday (5/25) around 3:00 p.m., likely due to severe weather. We did not experience any other issues after this. 

• Experienced issues with temperature today (5/27) overnight around 1:00 a.m. We communicated with campus Facilities Management until roughly 3:00 a.m. Temperatures improved, but we will investigate further. 

 

OU:

• Occasional storage overload or compute nodes running out of RAM, otherwise smooth running
• Network monitoring almost there, still needs a few config modifications
• OURdisk mounted correctly now, but ticket on hold until I return from vacation at the end of the month