US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
1) thanks for the QR contributions. Did we miss anything?
2) Any updates to capacity spreadsheet pending?
3) future analysis systems facilities workshop this week will inform some of our activities going forward.
- Firstly, REANA deployment at BNL. Use k8s at BNL and containerized deployment. There is some dev work here to do, c.f. https://github.com/orgs/reanahub/projects/33
- Eric reports on-going work with REANA-slurm
- Eric also reports availability of OpenShift (Enterprise RedHat Kubernetes) they have at BNL that could be used for protopyping
- Kasuhik notes we need buy-in from Varena, WBS 5, to support users when it becomes production ready.
- Independently on IRIS-HEP SSL we'll explore deploying JupterLab, JupyterHub -to- Dask scheduling over HTCondor, deployed from Kubernetes.
- This is most similar to the UNL's AF (c.f. this talk and others)
- We may need to explore an Authz service compliant with CERN OIDC and e-groups, eventually.
- (This was suggested by Brian B after comments at the blueprint, to be investigated; a longer term project project - n.b. we already have CILogon-to-CERN OIDC for identity, but not the authorization part.)
- A related related IRIS-HEP SSL activity is to build a federated Kubernetes (multi-cluster) with a number of contributing sites (UC, UNL, UW, UCSD). The second meeting of this group is next Tuesday.
- Likely management will want to weigh-in, so to be discussed.
5) Second k8s-hep meetup is Dec 1,2 https://indico.cern.ch/event/968726/ - a technical meeting for sharing Kubernetes knowledge in the HEP context.
6) We need to ramp up the devOps team for frontier-squid now that we have a slate-registered Kubernetes cluster at each Tier2 and Frontera:
- MWT2
- NET2
- TACC
- SWT2-CPB
- AGLT2
- This should be quite simple. How can we get this organized? (c.f. https://docs.google.com/presentation/d/1hb5MqUQDGtkxbhC9akEflCC9wHx0k0E1KwRfQ9qI_ZA/edit?usp=sharing)
- Ofer will organize meeting with Judith, Ilija, Rob
7) Data lakes meeting again this afternoon (Shawn organizing) Meeting notes at https://docs.google.com/document/d/1HII7CpcIJP3uyhlZahLt6L2qI7dG_JDkMBWa7OpQ2GE/edit
- Firstly, REANA deployment at BNL. Use k8s at BNL and containerized deployment. There is some dev work here to do, c.f. https://github.com/orgs/reanahub/projects/33
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Lin (University of Wisconsin), Matyas Selmeci
-
13:20
→
13:35
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
-
13:20
TBD 15m
-
13:20
-
13:35
→
13:40
WBS 2.3.1 Tier1 Center 5mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
-
13:40
→
14:00
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))- Good running with very few tickets/outages over the past 2 weeks.
- No need for plots this week.
- I would like everyone to clean up and remove unneeded panda queues and decommissiond clusters. In validating the accounting data, I got confused by 2 similarly named clusters one of which was no longer in service,
- I find the hodge podge of site names confusing so I made a strawman proposal for naming the sites so they show up together in CRIC and Panda Monitor to a few of you:
- US-ATLAS-AGLT2
- US-ATLAS-T1
- US-ATLAS-MWT2
- US-ATLAS-NET2
- US-ATLAS-SWT2-OU
- US-ATLAS-SWT2-CPB
- US-ATLAS-SWT2-UTA
- We have been asked by Julia Andreeva to make a presentation on November 22 at the WLCG/HSF "...to give a presentation summarizing US ATLAS T2 experience or probably representatives of a particular site would like to share their experience/plans?"
-
13:40
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)
No ticket
- had one file access problem flagged to us last Friday, but files exist, believed resolved.
Hardware
- 3 new VMware nodes for UM
Powerful, 512G RAM, 256 HT, 6.4TB NVME, 25G NIC
Racked and ESXi installed.
Need further cabling and configuration to be put into production.- MSU new storage online (472 TB usable)
retired 2 TB disks from ~2010 (300 TB)
currently migrating data for retiring one MD3060e with history of problems
will re-locate the 60x 3 TB disks from the MD3060e
Services- Scheduling OS upgrade our main switches (Juniper EX9208) at both sites
Next Tuesday 03-Nov-2020
Should only involve 2-3 ~15mn reboots at both sites and no loss of jobs
will not set site offline, just at risk: intermittent outage- Adding anti-spoofing filters to our main switches
to avoid abuse of our DNS servers
by spoofing source address to come from our subnets -
13:45
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))
-
13:50
NET2 5mSpeaker: Prof. Saul Youssef (Boston University (US))
Smooth operations with a full site for the past two weeks.
SLATE xCache working.
Restarted space token dumps at the request of ADC.
One of the nics on a GPFS NSD was failing and needed to be disabled.
Usual low level of old CPUs failing with temperature alarms or failing disks.
Lots of NESE-related planning happening.
On our agenda: OSG 3.5, HTTPS-TPC enabling, ipv6
- 13:55
- Good running with very few tickets/outages over the past 2 weeks.
-
14:00
→
14:05
WBS 2.3.3 HPC Operations 5mSpeakers: Doug Benjamin (Duke University (US)), lincoln bryant
debugging going slowly for ALCF Theta. Task migrated from NERSC is not starting. (gets killed by Harvester)
Networking issues at TACC sorted out and testing to run multiple pilots per compute node resumes
NERSC ran out of Inode quota causing Sample A jobs to fail. fixed and work is resuming. got another 5M hours for NERSC.
BNLHPC_DATADISK storage is full. Files need to be removed from it.
-
14:05
→
14:20
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:05
Analysis Facilities - BNL 5mSpeaker: William Strecker-Kellogg (Brookhaven National Lab)
-
14:10
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
- 14:15
-
14:05
-
14:20
→
14:40
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind
- Updated Facilities Services spreadsheet with SLATE deployment status and highlighting areas of concern
- Reminder about OSG 3.4 EOL in November
- Status of new BNL Site definition for VP?
-
14:20
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
-
14:25
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
Analytics
- ES upgraded 7.9 - working on Auditing
- Wrote deduplicator for perfSONAR.
- Working on Spark K8s deployment at CERN. Needed for DDM and popularity studies
XCache and VP
- SLATE and XCache working fine at all US sites.
- LRZ VP queue is back on - need to scale them back up.
- I still need to create an extra ATLAS site for ANALY_BNL_VP queue. Once done we will try to scale it up.
-
14:40
→
14:45
AOB 5m
-
13:00
→
13:10