US ATLAS Computing Facility

Wednesday 4 Dec 2024, 13:00 → 15:25 US/Eastern

Facilities Team Google Drive Folder

Zoom information

Meeting ID:  993 2967 7148

Meeting password: 452400

Invite link:  https://umich.zoom.us/j/99329677148?pwd=c29ObEdCak9wbFBWY2F2Rlo4cFJ6UT09

 

 

13:00 → 13:05 WBS 2.3 Facility Management News

Speakers: Alexei Klimentov (Brookhaven National Laboratory (US)), Dr Shawn Mc Kee (University of Michigan (US))

TIM : please book rooms before Dec 5

New WLCG T1 in Serbia

WLCG OB S.Camapana

The 2025 numbers could be ~50% higher than planned (and resourced). Remember that 4 weeks were “moved” from 2025 into 2024, so 2025 was expected to be 120 fb-1 – 4 weeks for ATLAS and CMS

The WLCG priority should be ensuring that the experiments can collect and archive data (storage, particularly tape). This might come at the cost of something else (e.g. CPUs and in some respect disk)

We started the discussion at the management Board. We will re-iterate in January.

13:05 → 13:10 OSG-LHC

Speakers: Brian Hua Lin (University of Wisconsin), Matyas Selmeci

• OSG 23 + 24 releases
  
  • XRootD 5.7.2 security release: built into the OSG, undergoing integration testing
  • HTCondor 24.2.2: fixes a critical issue that could result in data loss for misconfigured startd's. Critical issue only affects 24.2.1
• 100% CPU + memory utilization project goals? Target timelines?
• Monthly ATLAS + HTCSS meeting (some Wednesday @ 2pm)
• Support coordination in the post-GGUS world?

13:10 → 13:30 WBS 2.3.1: Tier1 Center

Convener: Alexei Klimentov (Brookhaven National Laboratory (US))

13:10 Tier-1 Infrastructure

Speaker: Jason Smith

• OpenShift: network software bug fixed in 4.17.7, performing upgrade now, need another week or so of testing before we can declare this production

• Internal config management infrastructure upgrade in progress, Puppet gitea server upgraded, Puppet and Foreman are next

13:15 Compute Farm

Speaker: Thomas Smith

13:20 Storage

Speakers: Carlos Fernando Gamboa (Department of Physics-Brookhaven National Laboratory (BNL)-Unkno), Carlos Fernando Gamboa (Brookhaven National Laboratory (US))

Operations Update:

• HPSS upgraded to version 8.3.u28 (12/03-04).

• dCache HPSS client versions upgraded.

• Successful intervention: tape-related endpoints are being re-enabled.

• Memory upgrades on 11 pool hosts are expected to be completed today, with 6 out of 11 already finished. The upgrades are being performed in a rolling fashion, one node at a time.

Dual "Home" Pool Tests:
The configuration of dual "home" pool tests on the test instance is ongoing. Preliminary tests have identified the following:

• An issue to be reported to the dCache developers, along with an alternate workaround.

• PoolManager workflow configuration updates.

• Pool’s network routing changes.

Additional Work:

• Extended monitoring capabilities.

• Improved configuration management.

13:25 Tier1 Operations and Monitoring

Speaker: Ivan Glushkov (Brookhaven National Laboratory (US))

Since last report (11/27/2024):

• Transfer failures due to remote storages - AGLT2, DESY-HH

• Slightly lower occupancy (97% instead of 98%)

  • Reducing the meanrss and

  • looking into failing wrappers. Wrapper on some nodes were restarted by the local condor which makes Harvester killing the wrappers. This means that there were ~25% less wrappers for BNL to run jobs in. This controbuted to the lower (by 1%) occupancy of the site.

13:30 → 13:40 WBS 2.3.2 Tier2 Centers

Updates on US Tier-2 centers

Conveners: Fred Luehring (Indiana University (US)), Rafael Coelho Lopes De Sa (University of Massachusetts (US))

• Good running over the last 30 days.
  • AGLT2 was partially drained Nov 19-23 due to cmvfs issues
  • AGLT2 has suffered job failures since Nov 25 due to an only partly successful update of their dCache nodes RHE9
  • MWT2 suffered job failures on Nov 28-29 for some sort of high memory usage.
  • SWT2 OU had a 1 day downtime.
  • SWT2 CPB has had various issues that cause job failures: older nodes failing, transfer issues, storage issues etc.
• The struggle with cvmfs continues with AGLT2 being the most affected.
  • On several occasions AGLT2 and MWT2 experienced problems accessing cvmfs from a large number of servers but only AGLT2 had servers hang as a result.
• There have been high memory jobs and MWT2 is experimenting with gracefully terminating the jobs using cgroups/condor.
  • Looks like the upcoming release for condor 23 of condor 24 must be used for EL9 systems. We are going to update a group of older servers OSG 24 and see if it helps.
  • Still working to understand why the jobs are being killed (SIGKILL) instead of gracefully (SIGTERM) at least on EL9.
    • Going to OSG 24 may solve this. 
• I am banging on understanding the in and out of warranty compute and storage capacity after all for the sites AFTER all current purchase are online and retirements have been made.
  
  • I have updated the "v70 Dec 24" tab of the capacity sheet to allow sites to enter ordered but not in production equipment in separate columns.
  • I will add functionality to calculate the average memory per core. I developed this on a private version of the capacity sheet that does this and I will add the functionality to the main capacity sheet.

13:40 → 13:50 WBS 2.3.3 Heterogenous Integration and Operations

HIOPS

Convener: Rui Wang (Argonne National Laboratory (US))

13:40 HPC Operations

Speaker: Rui Wang (Argonne National Laboratory (US))

Jobs are running smoothly at both sites

TACC: AY25 started (143K SU)

• 110K SU used in AY24 (mainly during Sept-Nov)
• Maintenance on Dec 10 and TACC exscale day follows
  • Downtime scheduled for BNL user data migration

Perlmutter: 130K CPU + 40 K GPU hour remains

13:45 Integration of Complex Workflows on Heterogeneous Resources

Speakers: Doug Benjamin (Brookhaven National Laboratory (US)), Xin Zhao (Brookhaven National Laboratory (US))

working changing globus connect server v5.4 authentication/authorization to beable to use backend dCache storage instead of lustre.

13:50 → 14:10 WBS 2.3.4 Analysis Facilities

Conveners: Ofer Rind (Brookhaven National Laboratory), Wei Yang (SLAC National Accelerator Laboratory (US))

13:50 Analysis Facilities - BNL

Speaker: Qiulan Huang (Brookhaven National Laboratory (US))

13:55 Analysis Facilities - SLAC

Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))

14:00 Analysis Facilities - Chicago

Speaker: Fengping Hu (University of Chicago (US))

Maintenance Completed Before Thanksgiving Holiday

• Performed OS and firmware updates on all nodes.
• Upgraded Kubernetes from v1.25 to v1.28.
• Updated Rook from v1.10 to v1.12.
• Upgraded Calico from v3.26 to v3.28.

Elastic Computing In the works

• Bursting to MWT2:
  • Working on implementing glideins (docker universe pilot jobs) to join the AF Condor pool.
  • glideins workers will have the same enviroment as the local AF workers - access to the same file systems, including /home and CephFS /data.

 

AF images

• New Julia image available.
• All new ml-platform image available for testing.

14:10 → 14:25 WBS 2.3.5 Continuous Operations

Convener: Ofer Rind (Brookhaven National Laboratory)

• • Network load testing for NET2 and SWT2 is on hold until at least next week;  BNL tests will be run tomorrow

14:10 ADC Operations, US Cloud Operations: Site Issues, Tickets & ADC Ops News

Speaker: Ivan Glushkov (Brookhaven National Laboratory (US))

 

• Tickets:
  • Opened USATLAS GGUS tickets (Link)
    • BGP Tagging: AGLT2 (GGUS:168397), MWT2 (GGUS:168404), NET2 (GGUS:168406), OU_OSCER_ATLAS (GGUS:168407), SWT2 (GGUS:168409),
  • AGLT2: Destination overwrite errors. DB problem after upgrade? Ongoing (GGUS:169084, dCache:10675)
  • SWT2: GoeGrid - SWT2 failing transfers. IPv6 issue outlide of SWT2 campus. Ongoing (GGUS:168756, GGUS:168751)
  • WT2: Not in GGUS since all services ware removed from OSG topology
• ADC Ops:
  • GGUS helpdesk: Please test it (Link). We need a statement from ATLAS side from the WLCG Ops Coordination meeting tomorrow
  • 5B MC events foreseen to be processed during the Christmas period
  • Removed production role for 70 users. If needed, please contact Petr Vokac.

14:15 Services DevOps

Speaker: Ilija Vukotic (University of Chicago (US))

XCache

a lot of issues at AGLT2, LRZ and Oxford. Build and installed everywhere 5.7.1

Reliability seems to be better since then.  There is still one xcache node at AGLT2 (managed through NRP) that shows problems.

VP

VP service works nominally

There was an issue in Pilot time monitoring causing file-exist checks not to be accounted for. After that was fixed we found some VP jobs were experiencing very slow file-exist checks. Most drastic when BNL is origin. Not yet sure why it happens.

Rod is now testing a VP light approach - he adds remote but well connected RSEs to his VP queues for a read_wan and removes VP RSE. Jobs work fine.

Alessandra is now testing similar approach in UK.

Varnish

All varnish instances for both Frontier and CVMFS work fine.

Nov 26th we had a first test of Varnish infrastructure for CREST on HLT. There were issues with CREST origin server. Need to improve monitoring for the next test.

AF

Starting work that would allow AF AI assistant to do most things that a regular user could do. Then user could simply rule out an issue is on their side before bugging human support. eg. "I can't log in to login2, can you?" or "access to /data seems very slow, can you check?"

 

14:20 Facility R&D

Speaker: Lincoln Bryant (University of Chicago (US))

• Lincoln working on Kueue on stretched k8s
• Aidan working on Ceph/Rook items on stretched k8s
• UChicago team had a discussion with the SENSE folks to put MWT2/UC AF into the mesh. Synergistic with NET2 SENSE efforts. 
• Worked with Horst to update K8S at OU and fix a certificate issue. 

14:25 → 14:35 AOB