US ATLAS Tier 2 Technical

Wednesday 11 Dec 2024, 10:00 → 11:00 US/Eastern

Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), Robert William Gardner Jr (University of Chicago (US)), Shawn Mc Kee (University of Michigan (US))

Meeting to discuss technical issues at the US ATLAS Tier 2 site. The primary audience is the US Tier 2 site administrators but anyone interested is welcome to attend.

10:00 → 10:10 Top of the meeting discussion

Speakers: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), Robert William Gardner Jr (University of Chicago (US)), Shawn Mc Kee (University of Michigan (US))

• Intermittent problems over the last 30 days.
  • Various instabilities at some sites and issues that we will discuss.
• The EL9 update drags on and on.
  • AGLT2 MSU still having problems with getting the officuak RHEL9 install to work.
  • MWT2 Illinois will move to a new building and when the servers come back they will be running RHEL9
  • SWT2 OU Updates done but still using an CentOS 7 container.
  • SWT2 CPB so close but not yet there/
• Next scheduled meeting on Dec 25 is canceled so will next meet on Jan 8.
  • I will discuss with Eric to see if we can find a better schedule.

10:10 → 10:20 TW-FTT

Speakers: Felix.hung-te Lee (Academia Sinica (TW)), Han-Wei Yen

1. Network connection with LHCONE through ESNet broke up during 14:10 10 Dec. - 16:16 11 Dec, about 26 hrs because of the failure of submarine cable.
2. Failure rate of event generation jobs increased in the recent two weeks but the efficiency is good. 
3. Working on the reconfiguration with backup link first to improve the network availability.
4. Reached 16TB traffic (in & out total) a day, >60TB a week in the last week of November. Hope the data volume will be more after all work nodes (2,200 cores) are online with Alma Linux 9.

10:20 → 10:30 AGLT2

Speakers: Daniel Hayden (Michigan State University (US)), Philippe Laurens (Michigan State University (US)), Shawn Mc Kee (University of Michigan (US)), Dr Wendy Wu (University of Michigan)

AGLT2 in downtime with 2 separate issues:

Another unexpected problem with transition of dCache spaceManager node head02 from EL7 to EL9. 
We now use version specific installs like postgresql15 and then use alternatives to pick the default version.
But ansible had not been updated to match this new method, and ansible updated the default version 13 provided by RHEL9 which broke the version selection.
This must have also been the source of the problem with backup/standby d-head02 that caused
the earlier replication problems and, indirectly, the intermittent outages.

While the replication node was offline and not processing files, one disk space was filling up.
Among other measures, we added some cleanup code to a script to delete some duplicated unrelated accounting files.
But, under some unexpected condition, the script could instead try to delete everything.
This dramatic misbehavior occured on a subset of nodes where this script runs regularly.

Current AGLT2 atatus and recovery:

We have declared a downtime that we will extend.

We have recovered head02 with a clean RHEL9 re-instal and using the last database backup from 06-Dec.
So there will be a loss of name records.

We are using this downtime to also upgrade/update head01 to RHEL9 and postgres15, then postgres17.
And we will also update head02 to postgres17.

We had a number of nodes that were damaged beyond repair and need rebuilding, including the DNS nodes at UM.

We are also considering upgrading the Condor nodes to RHEL9, but probably not yet OSG24.
We would benefit from the MWT2 knowledge and previous experience.

10:30 → 10:40 MWT2

Speakers: Aidan Rosberg (Indiana University (US)), David Jordan (University of Chicago (US)), Farnaz Golnaraghi (University of Chicago (US)), Fengping Hu (University of Chicago (US)), Fred Luehring (Indiana University (US)), Judith Lorraine Stephen (University of Chicago (US)), Lincoln Bryant (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US)), lincoln bryant

• Updating IU and UC compute nodes to condor 24.0.2 after discovering issues with cgroups v2 memory limits on 23.0.18
• All IU storage is in production. Retired the MD3460s at UC.
• UC Storage purchase is at the final phase.
• UIUC data center move and EL9 migration will happen from 01/01/2025 - 01/15/2025

10:40 → 10:50 NET2

Speakers: Eduardo Bach (University of Massachusetts (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US)), William Axel Leight (University of Massachusetts Amherst)

10:50 → 11:00 SWT2

Speakers: Horst Severini (University of Oklahoma (US)), Kaushik De (University of Texas at Arlington (US)), Mark Sosebee (University of Texas at Arlington (US)), Zachary Thomas Booth (University of Texas at Arlington (US))

SWT2_CPB: 

• Procurement

• We received the 12 storage nodes we ordered. 

• We have been developing a plan to deploy these, including plans for doing tests for EL9. 

• Likely will recruit the help of a student to physically move servers, while removing old servers. 

• EL9 Migration

• Resolved DNS issue with EL9 condor-ce. 

• Test jobs are running and completing successfully. 

• Currently creating a host group for K8 machines for Armen and picked two R410s to build as K8 machines. Will provide access to these machines to Armen for review.

• There is an issue with what path jobs are using as the scratch directory on worker nodes. We are looking into how to resolve this in HTCondor-CE 24. Other sites seem to have this issue as well.

• Plan on possibly adjusting the partition scheme for EL9 compute nodes if CVMFS cache location is changed. 

• Managing Certificates

• Updated certificates for two of our data transfer nodes. Verified the certs were correct and that restarting the service worked fine. However, we received transfer errors. Rebooting both nodes and starting services up seemed to have resolved issues. 

• Inventory Script

• We tested the inventory script from a student and discovered an issue. Student is working on resolving this before this coming Friday.

• GGUS Enable site network monitoring (GGUS 162991)

• Contact campus networking. 

• They shared Grafana plots that have the information we need. 

• They do not allow read-only SNMP queries for this data.

• I set up a web server to host the necessary information. 

• Lost access to the Grafana plots and contacted campus networking. 

• They opened a case and got this fixed, but now we need a way to query this data. I requested permissions with Grafana HTTP API or the ability to query the data source. They are working on this. 

• GGUS GoeGrid transfer issue (GGUS 168756)

• Reaching out to ESNET to check for issues. 

• Reaching out to ESNET to check for issues. 

• So far, we have seen any issues related to this at our site. 

• We believe the issue is upstream.

OU:

• Had power maintenance yesterday, otherwise nothing to report.