US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Lin (University of Wisconsin), Matyas Selmeci
OSG 3.5.31
- osg-ca-certs 1.94: adding the cross-signing Let's Encrypt root CA
- osg-release 3.5-5: preparation for separate OSG 3.5 upcoming repositories
- CVMFS 2.8.0
- XRootD 4.12.6
- SciTokens python library 1.3.1
Other
- OSG 3.6
- Draft container release policy
- Draft updated release repository for rolling 3.6 release
- Anyone use the "xrootd-status-probe" RPM from osg-contrib?
- NET2 "No Risk" outage in CRIC was due to downtime not matching CRIC-registered accepted protocols. To fix this in the future, make sure Topology registrations contain the appropriate services, then you can select them in the downtime.
- Looking for feedback on OSG site admin navigation reorganization (https://github.com/opensciencegrid/docs/pull/777). Requires github/docker to preview:
- Run the following commands:
$ git clone https://github.com/brianhlin/docs
$ cd docs
$ git checkout SOFTWARE-4441.reorganize-toc
$ docker run --rm -p 8000:8000 -v ${PWD}:/docs squidfunk/mkdocs-material:6.1.4 - Open 'localhost:8000' in your browser to view the documentation
- Provide any feedback to technology-team@opensciencegrid.org!
- Run the following commands:
-
13:20
→
13:35
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
-
13:20
FedOps for Frontier-Squid 15m
-
13:20
-
13:35
→
13:40
WBS 2.3.1 Tier1 Center 5mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
- Smooth operation in general
- dCache restarted yesterday, as a workaround for the transfer issues with Swiss sites caused by a bug in the CaNL library, triggered by a recent IGTF CA release.
- Thorough scanning and analysis on space usage in dCache was conducted, now the cause for the space reporting discrepancy is understood, biggest factor is duplicated file copies on the primary pools. The issue is fixed
-
13:40
→
14:00
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))- Reasonably stable running except NET2 where some sort of GPFS problem caused an outage:



- Network load on the CERN side - from today's ADC Daily minutes:
- Rucio
- (Martin) All the missing rules from the subscriptions problem created
- Transfers
- Transfers peak above 60GB/s due to Data Consolidation from above point
- Rucio
- Noticed at AGLT2 & MWT2.
- Requested for the Monit plots generated that TACC be moved out of MWT2 and into the US Tier 3 sites where the other supercomputers are.
- Also got the countries USA and United States of America combined into being just the USA
- Still need to wrangle with some other naming conventions.
- Still trying to solve stuck transfers issue.
- Agreed to help out Frontier-Squid Dev team
- Sysadmins should attend the ADC Weekly round table.
- Need to keep up with IPV6, HTTP-TPC, and OSG 3.5.
- Spend your network money by the end of February - that's a milestone.
-
13:40
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)
1 ggus ticket: 5 missing files. File were lost from hardware failure last Oct. They were not included in the original lost file we declared. Maybe because they did not have metadata in the chimera database.
1 ggus ticket about local muoncal jobs using the proxy server at CERN instead of aglt2 proxies, this is due to a bug in Frontier Server setup, Asoka fixed it.
Network issue:
2 weeks ago we reported a problem with the UM-MSU 2x40G link.
This was tracked down and solved by replacing one 40G transceiver in the MSU computing room AND rebooting the juniper line card hosting that port. The link was restored to full capacity 21-Jan-2021
Early this morning we had a new problem on the other 40G member of that same UM-MSU link. This seems to be resolved after resetting one transponder at the UM/Merit router interface.
Server room:
working on replacing the PDUs for some racks at UM site
Planning on replacing the WAN (replacing all WAN switches and reposition and recable machines) of UM Site. Placed order for Dell switches (9 S5232 and 9 S3048). Plan a week of downtime, maybe align with the moving time of MSU site(March or April), to rearrange the server room.
-
13:45
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))
UC
- Were seeing 3-4k open connections to the AGLT2 xcache a couple of weeks ago. After Ilija restarted xcache, things seem to be behaving better
- Applying necessary configuration changes to UC workers for migrating to SLATE frontier-squids
- UChicago SciDMZ egress maxed out 01 Feb
IU
- New head node equipment received, in the process of being built
- PO for new network equipment submitted
UIUC
- Transceivers ordered for the new compute nodes
-
13:50
NET2 5mSpeaker: Prof. Saul Youssef (Boston University (US))
We recovered from 1 week downtime of NET2_DATADISK, _SCRATCHDISK
Extensive & useful discussion at the ADC weekly yesterday. Multiple followups likely.
During the downtime, we successfully ran using just NESE Ceph storage.
Lincoln and Ilija are helping us run squids on our SLATE node (re: low priority DDM ticket).
-
13:55
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
OSG and XRootD software updates progressing:
-- working on these in our test cluster
-- lost a couple of days - updating our local Tier-3 cluster, now doneNew perfSONAR nodes have been ordered - awaiting delivery
Met with campus networking personnel to discuss IPv6 implementation:
-- they are making progress
-- a few issues they still need to address (DNS, etc.)
-- decision about the current switch in the data center that provides outside connectivity
-- we're setting up a host for testing purposesOU:
-- site running well, getting good 8-core opportunistic throughput as well
-- waiting on Dell quotes for new compute node purchases
- Reasonably stable running except NET2 where some sort of GPFS problem caused an outage:
-
14:00
→
14:05
WBS 2.3.3 HPC Operations 5mSpeakers: Doug Benjamin (Duke University (US)), Lincoln Bryant (University of Chicago (US)), lincoln bryant
NERSC -
Smooth running - getting lots of cycles see attached plot
Theta -
getting hours - see attached plot. Now have up to 10 jobs (384 nodes each for 9 hours) in the system.
If we don't use alot of hours in the next two weeks we will lose them due to ALCC pullback. US CMS is not running anything. so usage has to be all ATLAS.
Frontera:
- Occasional issues with $WORK (lustre) filesystem. Harvester cacher data nominally belongs here, but occasional "input/output" errors on this device. Files become non-deleteable by normal means (rm, find -delete ..) and Harvester cannot continue. Must file a ticket to get system admins to delete the file. This paused jobs for the last week or so as Lincoln didn't catch that this error had returned until yesterday. Will consider moving Harvester files back to $HOME or possibly $SCRATCH if issues continue.
- System now in downtime through the 8th for "texascale" (large scale) mode.
-
14:05
→
14:20
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:05
Analysis Facilities - BNL 5mSpeaker: William Strecker-Kellogg (Brookhaven National Lab)
-
14:10
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
- 14:15
-
14:05
-
14:20
→
14:40
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind
- AGIS to CRIC migration proceeding again, see Alexey's talk at Monday's TCB meeting
- Some residual operational issues during/subsequent to NET2 GPFS intervention
- Robust site topic discussion at the ADC weekly (US and UK)
- Discussion led to ideas for reconfiguration of PQs and SEs at NET2/NESE
- SLATE Frontier-squid deployment proceeding at NET2 (GGUS)
-
14:20
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
-
14:25
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
-
14:30
Kubernetes R&D at UTA 5mSpeaker: Armen Vartapetian (University of Texas at Arlington (US))
The computers I am using I borrowed from our admins, are several test machines usually used for evaluation of updates and services before they are implemented in the main SWT2 grid production cluster.
Right now the test cluster is utilized for such evaluation, and as part of it is undergoing complete rebuild, with possible hardware changes. Once I get it back in 1-2 week, I'll reinstall k8s, and proceed with setting up k8s test queue.
-
14:40
→
14:45
AOB 5m
This afternoon is the monthly ESnet Blueprinting meeting for those who want to join.
Meeting Notes: https://docs.google.com/document/d/1zng0gXq-OsPyw0km0JygVDQM0ZQK806jXOQcACTURlQ/edit
Eli Dart is inviting you to a scheduled Zoom meeting.
Topic: Eli Dart's Zoom Meeting
Time: Feb 3, 2021 11:30 AM Pacific Time (US and Canada)
Join Zoom Meeting
https://esnet.zoom.us/my/elidart?pwd=dmx6TGZEalBENmN1L3ZVVkllZy9NUT09
-
13:00
→
13:10