US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Lin (University of Wisconsin), Matyas Selmeci
-
13:20
→
13:35
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
-
13:20
TBD 15m
-
13:20
-
13:35
→
13:40
WBS 2.3.1 Tier1 Center 5mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
-
13:40
→
14:00
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))Lots of news:
- Each Tier 2 getting about $100k from NSF for FY21.
- Got generally full amounts for infrastructure/networking too... (Srini end of period money)
- Still waiting for Tier 2 review report.
- Generally good running for the last two weeks.
- AGLT2 traced transfer errors ipv6
- MWT2 still chasing low level of transfer failure. AC problem crept up at UC over last ~3 weeks.
- NET2 some low level of trouble with oldest systems
- SWT2 deletion issues at CPB
- Please benchmark all new compute servers as they are received.
- Slate tutorial needs to be scheduled
- Need to use Mattermost more
- Ofer get the info he needed about the mechanism for marking downtimes?
- CRIC migration planned to be completed in September see latest report (July 13): https://indico.cern.ch/event/936150/contributions/3933422/
-
13:40
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)
No open ticket
UM received all 2020 new purchased hardware, 5 R740x2d, 20 C6420, 80% of them are on the rack, and we are in the process of provision these nodes.
MSU bureaucracy was slower but POs issued, waiting on deliverydcache update to 5.2.25:
made us realize and find IPv6 problem, fixed.
still sporadic dcache problems with pools setting themselves offline after detecting consistency problems. Forwarded logfiles to dcache ticketafs: updated all AFS servers from SL6 to CC7, and afs server to 1.8.6
-
13:45
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))
UC
- Debugging temperature issues in the server room
- SE set offline over the weekend due to servers overheating and shutting down
- UIUC and IU workers fully online now, bringing UC up incrementally
- Equipment POs submitted
IU
- Equipment POs submitted
UIUC
- A couple workers down for ongoing hardware issues, otherwise running as usual
- Debugging temperature issues in the server room
-
13:50
NET2 5mSpeaker: Prof. Saul Youssef (Boston University (US))
Making final preparations for 6PB expansion of NESE_DATADISK space in NESE/Ceph. Existing NESE deployment has been running fine for many months now.
We're making the final preparations before ordering the first NESE Tape Tier purchase, deployment and launch. After extensive interviews with BNL, LANL, NERSC, TACC, NCSA, MIT, we've decided on a 9 frame, IBM TS4500 (same model that BNL is getting), with full capacity of 140PB. The front end will be a 1PB FC attached storage running IBM Spectrum Scale/Spectrum Archive ("GPFS"). Protocols will be POSIX & S3 via MinIO. I/O will initially be 11GB/s initially, ramping up to 17GB/s as we add more drives. MGHPCC has reserved an entire pod for this with space for a total of 4 libraries of this size.
Preparing for a major trip to MGHPCC to take care of quite a few backed up issues: Installing new workers, replacing fans, fixing broken disks, re-cabling and replacing a likely failing 15 year old TOR switch.
Operations have been smooth with the site fairly consistently full.
-
13:55
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
OU:
- Running well, no issue
- There were problems last week where arturos AP_REPR jobs were consuming MUCH more RAM than they had requested, which took down quite a few of our nodes because they ran out of RAM. That was apparently caused by scout jobs reporting incorrect (much too low) RAM usage on these jobs, which was eventually corrected. Paul and others are investigating how to prevent this from happening in the future.
- Updates from last time:
- OSG downtime only causes Switcher to turn sites of if OUTAGE is chosen, not SEVERE or other options.
- SAM tests have been modified to only request 10 minutes of walltime now.
UTA:
=> Hardware from the most recent round of purchases is being delivered.
=> 8/4 - 8/5: Filesystem mount on one of the gridftp hosts hung up, causing file deletion errors. Fixed - closed the ggus ticket.
=> 7/26: Problem with the chilled water supplying the CRAC's at SWT2_CPB. Facilities responded quickly - A/C's returned to normal temps after ~one hour.
=> Otherwise smooth operations over the past two weeks.
- 14:00 → 14:05
-
14:05
→
14:20
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:05
Analysis Facilities - BNL 5mSpeaker: William Strecker-Kellogg (Brookhaven National Lab)
-
14:10
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
- 14:15
-
14:05
-
14:20
→
14:40
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind
-
14:20
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
-
14:25
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
SLATE
deployment at OU: Patrick will continue with it after coming back from vacation (10th Aug).
NET2 Saul should be working on it now.
VP
Still in broker-off as the release on 97a is still not there. (Ed still on vacation)
Work on "fully diskless" PQ still blocked waiting for explanations from Tadashi/Fernando (for few months already).
XCache
using OSG base at 5.0.0-1
Analytics
running fine
-
14:20
-
14:40
→
14:45
AOB 5m
-
13:00
→
13:10