US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn Mc Kee (University of Michigan (US))
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Hua Lin (University of Wisconsin), Matyas Selmeci
-
13:20
→
13:50
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
- 13:50 → 13:55
-
13:55
→
14:15
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))-
13:55
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn Mc Kee (University of Michigan (US)), Prof. Wenjing Dronen
- 14:00
-
14:05
NET2 5mSpeakers: Prof. Rafael Lopes de Sa (University of Massachussets, Amherst), William Axel Leight (University of Massachusetts Amherst)
NET2.0: Problems with transfer to NESE. Augustine Abaris from BU investigating NESE DTN. Problem persists but it is mitigated after reducing the number of jobs running.
NET2.1: Rack 1 space at MGHPCC freed up. Working on ESnet connection. Best strategy for this year is to do 10GB/s via NoX. We are trying to plan for 100 GB/s (with equipment capable of 400GB/s) for next year via NEREN. Partial inventory of NET2.0 equipment made with help of SBU, hoping to complete the full inventory with help from BU and UMass RC teams. Some glaring discrepancies with US-ATLAS facilities spreadsheet that we are trying to understand. Discussions about NESE EC configuration on-going on MM.
-
14:10
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
OU:
- Running well in general.
- There was a network issue at OneNet in Tulsa which impacted inbound transfers to OU last week. Fixed.
- This caused temporary draining of site.
UTA:
- Ongoing work in K8s, job mix balance, Need to look at accounting for APEL
- Problems with trying to create new SLATE development host; OSG is working on it
- Cluster running smoothly
-
13:55
-
14:15
→
14:20
WBS 2.3.3 HPC Operations 5mSpeakers: Lincoln Bryant (University of Chicago (US)), Rui Wang (Argonne National Laboratory (US))
-
14:20
→
14:35
WBS 2.3.4 Analysis FacilitiesConveners: Ofer Rind (Brookhaven National Laboratory), Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:20
Analysis Facilities - BNL 5mSpeaker: Ofer Rind (Brookhaven National Laboratory)
- Tom Smith's updates have been merged into the official afmetrics_collector package: https://github.com/maniaclab/afmetrics_collector (dashboard). Ready for SLAC to try out?
- Ongoing testing of ServiceX on OKD
- HSF AF Forum tomorrow
- Benchmarks for ACAT?
-
14:25
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:30
Analysis Facilities - Chicago 5mSpeakers: Fengping Hu (University of Chicago (US)), Ilija Vukotic (University of Chicago (US))
- we improved our jupyterlab interface for GPU selections. Previously users were given GPU selection by memory size but can't see the availability information, so notebooks can endup in waiting for a very long time.
- added cvmfs health monitoring in troubleshooting intermitant cvmfs availability issue on htcondor pods
- we are helping two demonstrations for PyHEP (one was this morning, next tomorrow morning)
-
14:20
-
14:35
→
14:55
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind (Brookhaven National Laboratory)
- NET2.1 Migration discussions on Mattermost (Ceph and Infrastructure)
- Augustine working on NESE storage server load issues
- XCache prefixing issue in Rucio, status?
- Interesting talk by Rod Walker at this morning's GDB meeting about preparing for and dealing with expected spike in European energy costs, likely starting in January
- Federated DevOps meeting: Notes
-
14:35
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeaker: Mark Sosebee (University of Texas at Arlington (US))
-
14:40
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
- XCache
- all running fine
- will be testing v5.5 next week
- VP
- Rucio issue made xcache paths not prepended thus disabling all xcaches. VP queues still worked but read directly from origins.
- Fixed last night and now things are back to normal.
- Analytics
- Now we have all of the Squids monitored in UC ES.
- Today we will have all of the xAOD access snooping data indexed back in ES. All 7 years of data.
- ServiceX
- Today we should have v1.0.31
- Will install have it installed in preproduction today and if everything is fine on AF on Monday.
- XCache
-
14:45
Kubernetes R&D at UTA 5mSpeaker: Armen Vartapetian (University of Texas at Arlington (US))
- The K8S cluster was stable past weeks running production jobs.
- One change in the configuration in CRIC was to limit SCORE jobs, after SCORE jobs gradually occupied most of the cores. After the limit was in place we were back to reasonable mixture of SCORE/MCORE jobs. To understand how to optimize the limits and maybe the method itself.
- Monday/Tuesday we had some nodes in stuck state (didn't happen before). Looks like some nodes run out of memory. Just restarting helped, and today all looks fine, but this needs some further investigation to understand the root cause.
- NET2.1 Migration discussions on Mattermost (Ceph and Infrastructure)
-
14:55
→
15:05
AOB 10m
-
13:00
→
13:10