US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Lin (University of Wisconsin), Matyas Selmeci
OSG 3.5.25 (tomorrow)
- CVMFS 2.7.5
- Frontier Squid in EL8
Miscellaneous
- European HTCondor week next week (Mon-Fri 8-11am https://indico.cern.ch/event/936993/)
- OSG 3.4 EOL at the end of November!
- Expect to hear from us to make sure your CEs are updated to HTCondor-CE 4!
- Documentation for upgrade to 3.5: https://opensciencegrid.org/docs/release/release_series/#updating-to-osg-35
- EL 8 WN software available: CVMFS, osg-wn-client in release, squid in EL8 to go out this week
-
13:20
→
13:35
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
- 13:20
-
13:35
→
13:40
WBS 2.3.1 Tier1 Center 5mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
-
13:40
→
14:00
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))- Good running until recently.
- 3 GGUS tickets open
- Storage problems today at AGLT2, MWT2, & NET2 (Minor CPB issues solved a few days ago)
- Putting HTTP-TPC as the first protocol at MWT2 on hold
- SLATE server at all sites milestone nearly done
- Infrastructure/network funding? Soon I guess....
- ipv6 status?
-
13:40
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)
Tickets:
148265: dcache storage went offline, The problem was traced to two of our zookeeper instances having filled their var partitions thus losing quorum. We added compression to their logrotate
and that should prevent the problem from re-occurring.
14829: dCache Transfer failing, this is caused by one dcache pool node having timeout transfer), we replaced the link between 2 main switches, which significantly improved the transfer rate to this pool node.
148666: dCache low transfer efficiency. One of the pool nodes lost one of the system disks, it crashed at trying to reboot itself overnight.Hardware:
All UM work nodes and dcache pool nodes from 2020 new purchase are in production. MSU nodes are still in process.Software update:
www.aglt2.org and wiki rebuilt and re-allowed through campus firewall
squid-frontier to 4.13-1.1
check-mk 1.6p17
ELK cluster 7.9.1, added 4 powerful nodes to the cluster. (40Gpbs link, 1TB RAM, 2*3TB Nvme cards)7 AFS servers all migrated from SL6 to CC7.
-
13:45
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))
Multiple dcache issues trying to update to 5.2.28 so we downgraded and are sitting on 5.2.25
One of our storage machines had a controller crash over the weekend. We brought back up and there haven't been issues since.
Missing files GGUS ticket: 148601 we are investigating to try and pinpoint the cause. Brought this up with the dcache team.
Errors have dropped back down and we're in working condition again.
UC:
- Waiting on several new compute nodes to be delivered and getting our last bit of misc. hardware necessary to start getting our new compute online.
- Storage issues. Working on pinpointing and fixing.
IU:
- Currently building new compute. Over half is built and in production.
- Getting compute on IPv6 in waves
UIUC:
- Working on getting down compute back up and running.
- 13:50
-
13:55
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
UTA:
- Deployed 1.5 PB of storage, final testing on another 1.5 PB, more storage to come
- Will retire very oldest storage (MD1x000 based systems): ~600TB
- Awaiting internal UTA delivery of latest compute nodes, maybe today
- Deployed SRR reporting script for SWT2_CPB
- Waiting to hear if the JSON is acceptable.
- Will deploy to UTA_SWT2 and OU
OU:
- Nothing to report, all running well.
- Getting great opportunistic throughput right now, up to 1000 additional slots.
- Deployed 1.5 PB of storage, final testing on another 1.5 PB, more storage to come
- Good running until recently.
-
14:00
→
14:05
WBS 2.3.3 HPC Operations 5mSpeakers: Doug Benjamin (Duke University (US)), lincoln bryant
-
14:05
→
14:20
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:05
Analysis Facilities - BNL 5mSpeaker: William Strecker-Kellogg (Brookhaven National Lab)
-
14:10
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:15
Analysis Facilities - Chicago 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
-
14:05
-
14:20
→
14:40
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind
- SLATE deployed at BU last week. Status of Xcache?
- Patrick and Lincoln working on SLATE install at CPB
- HTTP-TPC changeover at MWT2 still being held up by lingering problems after dCache upgrade
- Some discussion at yesterday's ADC weekly re: best use of new upcoming NESE tape storage and how the storage element will be accessed (need HTTP-TPC solution)
- Mark, with Fred and Ofer, putting together a facility-wide service profile spreadsheet (sw versions, capabilities)
-
14:20
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
-
14:25
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
- Ilija presented a nice overview of Analytics and ML status at yesterday's ADC weekly.
-
14:40
→
14:45
AOB 5m
-
13:00
→
13:10