US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn Mc Kee (University of Michigan (US))
The next HEPiX meeting will be April 25-29, 2022. Please consider putting in abstracts in any of a number of tracks. https://indico.cern.ch/event/1123214/abstracts/ While site reports are an option, we are involved in many other areas appropriate to report on for HEPiX.
The #48 LHCONE/LHCOPN meeting concluded today. Agenda and talks are available at https://indico.cern.ch/event/1110783/
Please get your quarterly reports in by Friday, April 15.
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Hua Lin (University of Wisconsin), Matyas Selmeci
-
13:20
→
13:50
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
-
13:20
Postponed till April 13: Storage Consistency Checks at the Tier1 15m
Talk is postponed till our April 13, 2022 meeting.
Speaker: Shigeki Misawa (Brookhaven National Laboratory (US))
-
13:20
-
13:50
→
13:55
WBS 2.3.1 Tier1 Center 5mSpeakers: Doug Benjamin (Brookhaven National Laboratory (US)), Eric Christian Lancon (Brookhaven National Laboratory (US))
-
13:55
→
14:15
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))-
13:55
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn Mc Kee (University of Michigan (US)), Prof. Wenjing Wu (University of Michigan)
Received and installed the 8 R740xD2 storage nodes for dCache (5 at UM, 3 at MSU), put them in production and rebalanced all the pools. This adds 2.8TB usable space, total usable space token is 10PB.
03/19/2022
The UM Tier3 building had a scheduled power outage, and it was not supposed to affect the status of the Tier3 server room, however water leakage happened in the morning, and the water cooling system stopped working, the server room got overheated, we shut down all work nodes in this room. This problem was resolved in 3 hours(the plumber fixed the leakage, and the clean crew cleaned the water on the floor), and we brought all work nodes back online. Fortunately no damage was made during this incident.
Ongoing ticket with Dell support for R6525. PERC would sometime not show up after boot. Traced to PERC data cable at motherboard end. The cable is very close to a metal post and a lid latching hook. In our case it must have been partially damaged during assembly. Be careful when you open (and close) the lid that the cable is not in the way of the post.
-
14:00
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Jess Haney (Univ. Illinois at Urbana Champaign (US)), Judith Lorraine Stephen (University of Chicago (US))
Intermittent site-wide draining over the past couple of weeks. Still investigating.
UC
- The last phase of the server room relocation project was this week. All MWT2 equipment is now in its new data center location. The last four (relocated) storage servers were brought back online. Currently rebalancing datasets over all pools.
- Five storage nodes offline today/yesterday due to temperatures (GGUS 156668). The issues is the inlet temperatures sometimes become elevated due to air circulation. We are working with the facility manager to address this to find a permanent solution.
IU
- Moved IU gatekeeper from UC to IU. Also increased the number of max running jobs on the IU gatekeeper
- FY22 compute ready to be put online
UIUC
- FY22 compute arrived and is in the process of being installed
-
14:05
NET2 5mSpeaker: Prof. Saul Youssef
Two (?) GGUS tickets...
o Forgot to renew cert.
o CRL failed to update on xrd-gpfs cluster for unknown reason.
Both fixed.
Run 3 preparation:
1. perfsonar & ipv6 & 88 workers installed but not online yet
2. OSG 3.6
NESE Ceph
New rack arrived except for Cisco top of rack switches.
NESE TapeFixed Harvard networking issue. Dimitrios & Xin helping 50 PB ATLAS space commissioning.
NET2 expansion
Major progress. -
14:10
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
UTA:
- UTA_SWT2 Retirement
- Cluster is powered off
- Equipment being moved to campus on 4/1
- Cleanup of CRIC/OSG Topology on 4/1
- SWT2_CPB
- FY22 storage racked (3.2PB usable) burn-in testing ongoing
- gk10.atlas-swt2.org is new CE (from OSG 3.5-upcoming)
- Moving old CE to OSG 3.6
- Test panda queue created for LSM retirement;
- Works with current Rucio/CRIC configuration using gsiftp
- Configuration for internal XRootD door broke LSM in standard queue
- Problem understood; more testing required
OU:
- Running well, no issues
- Had release 22 MCORE jobs running compute nodes out of memory last week, but that was solved by reducing maxrss in CRIC
- Upgraded SE and backend storage to xrootd 5.4.2 from osg35-upcoming
- Working on bringing up new CE GK to install OSG 3.6 on.
- Still waiting for some of the purchased compute nodes from Dell
- UTA_SWT2 Retirement
-
13:55
- 14:15 → 14:20
-
14:20
→
14:35
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:20
Analysis Facilities - BNL 5mSpeaker: Ofer Rind (Brookhaven National Laboratory)
- BNL A100 installation proceeding: CUDA is set up, burn-in tests running.
- Benchmarking with MIG shows ~15% performance hit - plan to run this way on some of the GPUs
- Presentations at last week's HSF AF Forum Kick-off meeting.
- AF discussion at next Thursday's Snowmass CompF4 Topical Group Workshop
- BNL A100 installation proceeding: CUDA is set up, burn-in tests running.
-
14:25
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:30
Analysis Facilities - Chicago 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
-
14:20
-
14:35
→
14:55
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind (Brookhaven National Laboratory)
- One month from OSG 3.5 EOL
- Yet another major HC exclusion incident yesterday, this time due to problem with one of the Panda servers (aipanda047)
- Xin presented preliminary results from Tape Challenge today
- BNL SAM/ETF issues related to XRootd fallback protocol - not yet ready to test gridftp shutoff
- We are trying to organize a Snowmass pre-meeting discussion about Edge Services this Friday at 1PM ET
- DevOps: working on documentation for alert service and admins, need to make sure UTA decommissioning includes squid service (should be handled by topology update)
-
14:35
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
-
14:40
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
Squid:
- Continuing to work on a standard Ops document: https://docs.google.com/document/d/1ITYl1Z7yu9v3O7gKErMlCdlFT5h0iw-pDRspTBy1vsU/edit?usp=sharing
XCaches:
- all working fine
VP:
- working fine
ServiceX:
- a big reliability upgrade. Will be in the next release 1.0.30 hope to have it released tomorrow, tested and deployed in production by Monday.
- better logging
- new features getting tested
-
14:45
Kubernetes R&D at UTA 5mSpeaker: Armen Vartapetian (University of Texas at Arlington (US))
-
14:55
→
15:05
AOB 10m
-
13:00
→
13:10