US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn Mc Kee (University of Michigan (US))
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Hua Lin (University of Wisconsin), Matyas Selmeci
Release
- osg-scitokens-mapfile-6 with token issuer mappings for CLAS12, CMS, EIC, GlueX, IceCube, and LIGO
- XRootD 5.4.1
- osg-xrootd (base for OSG XRootD standalone, XCache) including a fix for X.509 auth over HTTP
- Install XRootD standalone: https://opensciencegrid.org/docs/data/xrootd/install-standalone/
- Install XCache: https://opensciencegrid.org/docs/data/stashcache/install-cache/
Other
- Make sure to keep your vo-client packages up-to-date! InCommon DN changes -> VO auth updates
- GridFTP hosts should be replaced with XRootD standalone: https://opensciencegrid.org/docs/data/xrootd/install-standalone/
- Container tag retention policy updated (images older than 6mo or security issues may be removed): https://opensciencegrid.org/technology/policy/container-release/#retention
-
13:20
→
13:50
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
-
13:20
AF, AGC, HSF, etc 15mSpeaker: Robert William Gardner Jr (University of Chicago (US))
-
13:20
-
13:50
→
13:55
WBS 2.3.1 Tier1 Center 5mSpeakers: Doug Benjamin (Brookhaven National Laboratory (US)), Eric Christian Lancon (Brookhaven National Laboratory (US))
-
13:55
→
14:15
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))- Good running... Some reduced availability at NET2 and SWT2.

- Need to check on the current status of IAM server support
- Missed WFMS meeting today was there any news?
- My spreadsheet of what we are working on to be ready for data taking:
- https://docs.google.com/spreadsheets/d/1KniOlqb4dbJ6dKUHBYYt9OfriKjhVpUqXguPvryIMY8/edit?usp=sharing
- Feel free to update items and items. Please do not delete items. Instead enter a comment that you won't do the item.
- We need to think about removing gsiFTP but it's still in use at some low level.
-
13:55
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn Mc Kee (University of Michigan (US)), Prof. Wenjing Wu (University of Michigan)
2/17/2022 (ticket 156003)
All queues of AGL2 site were set to TEST status, because most of the jobs were failing at rucio errors.
We found out that was caused by a change of the webdav endpoint of for the site.
We had changed the webdav endpoint from head01.aglt2.org to webdav.aglt2.org
with a Round Robin DNS entry including our 6 dcache doors.
We quickly realized the dcache doors did not have the right host certificates
(missing the subject alternative names in the host certs),
So we reverted the changes in ATLAS CRIC.
However, by design, rucio adds but does not automatically delete endpoints without a manual operation.
The rucio team was short handed, and did not process this manual operation in time,
so it caused a massive job failure and data transfer failure.2/18/2022
Resolved the problem of the scitoken not working on the gatekeepers
(the condor-ce variables AUTH_SSL_SERVER_CERTFILE and AUTH_SSL_SERVER_KEYFILE
were not configured to the right certificate and key files.
This error is shown in the log file condor-ce/Collector.log, under “will try to use 4096”)
We then configured all 4 gatekeepers to use sci token for authentication.Also upgraded our ATLAS test gatekeeper gate04 from OSG 3.5 to OSG3.6
(condor 9.0.9-1, condor-ce 5.1.3, only supports scitoken authentication)2/20/2022 (ticket 156060)
We installed new correct hosts certificates for webdav
and (re-)reconfigured ATLAS CRIC to use webdav.aglt2.org as the service endpoint for webdav protocol.
Then we got a new ticket about transfer failures due to one host certificate (dcdmsu01).
After some empirical debugging, it seems the cause is the “full chain” type certificate this host had.
In fact 5 of the 6 new certificates we received included the full chain of certificates,
which seems to be fine with most grid services, but not with webdav that failed authentication.
We truncated the certificate files to just the host cert and that solved the problem.2/25/2022
Nearly 1000 jobs failed because one of our ATLAS gatekeepers gate03 var partition was flooded
by the condor-ce AuditLog files. The default configuration in condor-ce is to keep 90 days of AuditLog,
on a busy gatekeeper node, an AuditLog file from one day can accumulate to 2GB.
To avoid this from happening again, reconfigure the value to 15 days (MAX_NUM_COLLECTOR_AUDIT_LOG = 15).
AMD Benchmark tests:
8x R6525 with AMD 7302 3GHz (64HT/system, 256G RAM)
Average HS06: 16.42
Average HEPscore: 17.7202-Mar-2022:
3x R740xd2 with 18 T disks from recent "end of cycle" order arrived at MSU .
That will be ~1000 TB usable for dcache.
We are still debating if and what we may retire. -
14:00
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Jess Haney (Univ. Illinois at Urbana Champaign (US)), Judith Lorraine Stephen (University of Chicago (US))
UC:
- Next phase of the data center migration is next week.
- Taking a downtime on Monday to migrate dcache database
- 40/80 of the compute from FY21 arrived. Other half soon
- Waiting to update to dcache 7 until after data center migrations
IU:
- Waiting on the new compute order to arrive. Should arrive this week and be installed next.
UIUC:
- Emergency maintenance today. Still running jobs from other gatekeepers
- Permanently shut off mwt2-gk
-
14:05
NET2 5mSpeaker: Prof. Saul Youssef
88 worker nodes arrived; testing.
New NESE storage and 10 DTNs arrived. Only waiting for Cisco switches.
Upgrading networking between NET2 and NESE.
GPFS 3TB pool extracted from file system; retired.
NESE tape commissioning with ADC, CMS.
Converging on plans for new bare metal cluster with new collaborators.
Other than bumps extracting 3TB pool, smooth operations. Site mostly but not always full. 1 GGUS ticket quickly fixed, but took many hours for HC to set us back online.
-
14:10
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
UTA_SWT2:
- Looks like the UTA_SWT2_DATADISK has been officially retired from Rucio as it no longer shows up as an RSE. Will follow up with DDM-OPS to verify that the disks can be turned off.
- Getting quotes for moving the racks back to campus and started making space for the equipment.
SWT2_CPB:
- Adding a second CE (gk10.atlas-swt2.org) based on OSG 3.5-upcoming. Configuration is ongoing but starting to work. Initially we will run Hammer Cloud jobs before moving production jobs to the new CE.
- Had a problem on the production CE that stopped all work. Problem is now fixed. Took the opportunity to rebuild compute nodes and get some configuration work straight for adding the second CE.
OU:
- Nothing to report, running well.
- Good running... Some reduced availability at NET2 and SWT2.
-
14:15
→
14:20
WBS 2.3.3 HPC Operations 5mSpeakers: Lincoln Bryant (University of Chicago (US)), Rui Wang (Argonne National Laboratory (US))
TACC
- Some recent failures, seems to be related to the payload itself rather than the HPC. Will contact the production folks if it persists.
- Still at 13% allocation remaining. Have 4 weeks left.
NERSC
- Cori running fine
- Perlmutter jobs running successfully in the test queue. Working on packing jobs similar to TACC, perhaps 16x 8-core jobs to be comparable to grid sites? Queue time limits also a concern. TBD.
-
14:20
→
14:35
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
- 14:20
-
14:25
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:30
Analysis Facilities - Chicago 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
-
14:35
→
14:55
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind (Brookhaven National Laboratory)
- xrootd multiuser plugin validation at BNL is complete (thanks Doug and Qiulan)
- FedOps: draft of operations guide under development.
- BNL VP jobs were not running. Discovered that repeated core dumps had filled the BNL xcache server boot volume - fixed. VP queue seems to be running ok since.
-
14:35
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
-
14:40
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
-
14:45
Kubernetes R&D at UTA 5mSpeaker: Armen Vartapetian (University of Texas at Arlington (US))
-
14:55
→
15:05
AOB 10m
-
13:00
→
13:10