US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
Just a heads up and invitation to attend next week's k8s-hep meetup, https://indico.cern.ch/event/968726/. Next week's WBS 2.3 Coordination meeting will be cancelled.
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Lin (University of Wisconsin), Matyas Selmeci
- XRootD 4.12.5 (for OSG 3.5 series) still in testing; no complaints yet
- XRootD devs say 5.1.0 will be out soon; if so, will skip 5.0.3 and go directly from 5.0.2 to 5.1.0 (for upcoming repos)
- HTCondor-CE, Gratia probe transition to EL 8 / Python 3 making good progress
- Expecting HTCondor 8.9.10 release next week (for upcoming repos)
Notes from Ofer:
- xrootd upgrade needd for http-tpc activation
- issue was whether we'd have a backport back to osg 3.5
- what is the deadline anyway? Alessandra: May.
- Feeling was we could get it done by May w/ 3.6 upgrade, which is sched for Feb.
- But concerns are heavy upgrade - so sites may not want to update right away
- So we still have quetions to work out - we need Brian
Wei - challenge in OSG 3.6 globus support gets dropped. So concerns that everything may not work out of the box. Would like OSG to support xrd 5 in osg 3.5. Both Ofer and Wei feel strongly about it.
Doug - claims you still need gridftp to still be there.
Can sites go to xrd 5 w/o osg 3.5; q for swt2 and net2.
-
13:20
→
13:35
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
-
13:20
New US LHC Analysis Facilities Forum 15mSpeakers: Doug Benjamin (Duke University (US)), Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Kaushik De (University of Texas at Arlington (US)), Mark Neubauer (Univ. Illinois at Urbana Champaign (US)), Mike Hance (UC Santa Cruz), Paolo Calafiura (Lawrence Berkeley National Lab. (US)), Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Verena Ingrid Martinez Outschoorn (University of Massachusetts (US)), Wei Yang (SLAC National Accelerator Laboratory (US)), William Edward Strecker-Kellogg (Brookhaven National Laboratory (US)), William Strecker-Kellogg (Brookhaven National Lab)
-
13:20
-
13:35
→
13:40
WBS 2.3.1 Tier1 Center 5mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
Normal operations in general.
REMINDER: Network intervention December 1st 6am-6pm.
Preliminary work on cost evolution and estimation of disk vs tape presented at the WLCG this week. Bandwidth requirements for HL-LHC make tape systems challenging. Work will continue, sharing infos. with FNAL.
Happy Thanksgiving!
-
13:40
→
14:00
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))- Good running with low error rates and very few GGUS tickets. For the last ~1 week here are the numbers of jobs finishing in the various possible states:
Site Done Failed Canceled Closed %Failed AGLT2 (no BOINC) 107160 3060 96 288 2% MWT2 241204 8736 197 306 3% NET2 93148 9053 24 5 8% OU_ATLAS 3492 25 2 0 0% OU_ATLAS_OPP 4289 23 4 89 0% SWT2_CPB 88990 9679 40 0 2% UTA_SWT2 21343 35 7 0 0% - Could everyone please report on how they are doing with their infrastructure/networking purchases.
- Already covered OSG 3.5.
-
13:40
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)
One Ticket :
GGUS ticket 149378: morphed into 2 issues.
- originally poor (~zero) transfer rate to UK
- recently : One file missing from our dCache.
Traced to day and pool with hardware failure.
Declared a list of files lost files but that file was missing.Services :
dCache updated from 6.2.5 to 6.2.10. Includes security fix. Smooth upgrade.
New AMD VMware cluster at UM now working well after simplifying network connections,
but with downside of more limited path redundancy. Had VMs in distributed vswitch losing connectivity.
Could recover by migrating to different host. Seemed to be tied to mixing uplink switches.New concern: Newer VMware versions not supporting Dell MD storage (iscsi at UM / direct attach at MSU)
current VMware version 6 seems to work, but version 7 may not... or cause support issues.
We are investigating new storage options.Hardware:
MSU retired 300 TB from last 2TB disks.
UM retired one problematic MD3060e storage enclosure (3TB*60 disks).
The controllers, power supplies, drawers can become spares.
Will move 60*3TB disks to recently freed MSU MD1200s to maitain storage capacity.UM old VMware cluster and old perfsonar nodes rebuilt into work nodes, adding 352 core.
MSU last purchase of C6420s in production (later today), adding 576 cores
-
13:45
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))
Debugging 1099 errors. Found that, for the compute nodes using XL710 chip and i40e driver on 10Gbps, upgrading the kernel from the stock 3.x series to the 5.x ml series, errors appear to have gone away for the past ~24 hours. Will upgrade 8 workers at IU and update to kernel-ml today and let run over the holiday weekend.
dCache upgrade scheduled for this coming Monday
Still investigating HTTP-TPC issues. It appears that the CRLs retrieved by running fetch-crl directly on the dCache machines differ for a couple of CAs in OASIS. Will move all dCache machines to use locally-installed CAs during Monday's downtime
New UIUC nodes expected to arrive next week
- 13:50
-
13:55
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
UTA:
1) All storage from most recent purchases is up, but putting the last few systems into production on hold for:
(i) Retirement decision about some MD3660i vintage storage
2) Resurrected 4 of 6 problematic compute nodes. Remaining 2 will require on-site support
3) Generally smooth operations
OU:
- Nothing to report, all running well
- Will probably have OSCER maintenance downtime Wed, Dec 16
-
14:00
→
14:05
WBS 2.3.3 HPC Operations 5mSpeakers: Doug Benjamin (Duke University (US)), lincoln bryant
-
14:05
→
14:20
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
- 14:05
-
14:10
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:15
Analysis Facilities - Chicago 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
Everything works fine.
Frontier platform needed quite a bit of updates. Now almost all of works. Working with Nurcan and Andrea on getting it fully out of Millisa's space.
New data source will be coming to ES: Each job perfmon data will be sent by pilot to UC logstash, indexed and analyzed in atlas-ml.org.
-
14:20
→
14:40
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind
-
14:20
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
-
14:25
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
XCache
creating special image to test issues with TLS between different versions of xroot servers.
creating a new 5.1.0 image to test new g-Stream.
VP
running at: BNL, MWT2, AGLT2, SWT2, NET2, LRZ-LMU, Prague2
making a new Prague_VP site as a diskless one. Waiting on Petr.
Performance is good and getting better.
Issues:
NET2 does not work as origin as it does not expose xrootd door
issue reading from SLAC
Need some developments that need to be agreed with Rucio people.
-
14:20
-
14:40
→
14:45
AOB 5m
-
13:00
→
13:10