US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Lin (University of Wisconsin), Matyas Selmeci
-
13:20
→
13:35
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
-
13:20
TBD 15m
-
13:20
-
13:35
→
13:40
WBS 2.3.1 Tier1 Center 5mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
- Running file in general, several dips in CPU usage due to ATLAS wide issues (CRIC)
- downtime of dCache is scheduled for August 24~25, for upgrade to v6.2
- for the ongoing reprocessing, some tails of staging, at the level of O(100), under investigation.
- singularity will be upgraded to 3.6 on the T1 computing farm, in a rolling fashion.
-
13:40
→
14:00
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))- Generally good running for the last two weeks but a couple of times Hammer Cloud took all US sites offline because of Rucio issues & the CRIC migration.
- AGLT2 struggled with the knock-on effects from the dCache issues.
- MWT2 still chasing low level of transfer failure. AC problem solved.
- NET2 some low level of trouble with oldest systems (older switches?). Enterprise switch issue took site offline briefly.
- SWT2
- Please benchmark all new compute servers as they are received.
- Slate tutorial after this meeting.
- Need to use Mattermost more
-
13:40
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)
New hardware purchase:
- UM put 3 dcache servers online, 2 other servers are about ready soon.
Put cables on all the C6420 computer nodes, ready for provision.
The PERC S410 cannot boot from RAID0. Considering options for best usage.- MSU received all purchased hardware, storage installation in progress.
Infrastructure
Power hickup at MSU on 11-Aug-2020.
Ground fault on lighting panel made it go offline
But misconfigured delays made upstream panel also go offline.
Our circuit is on generator backup, after ~20s.
Storage and critical networking is behind rack UPS, but not worker nodes.
For the future: we told that delays have been adjusted.
Some of our older managed PDUs didn't like the hickup, mainly the recovery.
We were were already planning on refreshing the older PDUs in the near future.Services
One dcache pool node was failing and shutting down some or all pools at once.
Have been tracing some errors through our network that we now understand is 100% correlated.
Working on replacing the cable between 2 main switches upstream.
This should be solved today, assuming and expecting this was the only problem.dCache went offline on last Sunday because two of our zookeeper instances
having filled their var partitions thus losing quorum.
We added compression to their logrotate.Also getting ready to upgrade to dcache 6.2, starting with upgrading to java11.
-
13:45
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))
UC
- First batch of compute and auxiliary servers arrived. Rest of compute delayed from Dell, arrival TBD
- Continuing to debug issues on one of the dCache storage nodes
IU
- First batch of compute arrived. Rest of compute also delayed, arrival TBD
- Continuing to debug IPv6 issues between IU and UC. Appears to be related to ACLs in place on the IU switches
-
13:50
NET2 5mSpeaker: Prof. Saul Youssef (Boston University (US))
Short downtime due to a hardware problem in one of the modules in the BU main N7K. Fixed over last weekend.Many old nodes repaired with replacement fans. 16 new worker nodes are racked, cabled and powered. TOR switch replaced on the oldest rack.
Preparing to recycle retiring worker nodes and 3TB sleds.
New NESE DELL storage is online, cabled with 25Gb/s x 2 and in Ceph. Will appear in NESE_DATADISK shortly.
Extensive preparations are underway for "Loch NESE" tape tier. Space for 133 PB, 50PB ATLAS buy-in, 1PB cache. Protocols: POSIX and S3.
Smooth operations with a full site otherwise. No tickets.
-
13:55
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
- Generally good running for the last two weeks but a couple of times Hammer Cloud took all US sites offline because of Rucio issues & the CRIC migration.
- 14:00 → 14:05
-
14:05
→
14:20
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:05
Analysis Facilities - BNL 5mSpeaker: William Strecker-Kellogg (Brookhaven National Lab)
-
14:10
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:15
Analysis Facilities - Chicago 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
-
14:05
-
14:20
→
14:40
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind
-
14:20
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
-
14:25
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
-
14:20
-
14:40
→
14:45
AOB 5m
-
13:00
→
13:10