US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Invite link: https://umich.zoom.us/j/91870138211?pwd=b2lLSURkSlhMNVpBRk1kWkRyM3IrZz09
-
-
12:00
→
12:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn Mc Kee (University of Michigan (US))
- 12:10 → 12:20
-
12:20
→
12:50
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
-
12:20
TBD 30m
-
12:20
-
12:50
→
12:55
WBS 2.3.1 Tier1 Center 5mSpeakers: Doug Benjamin (Brookhaven National Laboratory (US)), Eric Christian Lancon (Brookhaven National Laboratory (US))
-
12:55
→
13:15
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))- The last couple of weeks have been a good running period with fairly steady production.
- Several incidents caused by central services but ADC has gotten pretty good at catching them and restoring the sites to production in a few minutes.

- I am developing a spreadsheet to track Tier 2 site readiness for Run 3 data taking.
- I have info for all sites but have only entered AGLT2 and MWT2.
- OSG officially announced that the new EOL for OSG 3.5 is May 2.
- This allows a less hurried update while providing a deadline well before the start of data taking.
- Rob had an idea that we internally review the sites in preparation for the prescrubbing. This will get the information together that we need to present during the scrubbing process and also allow us to plan the FY22 purchases.
-
12:55
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn Mc Kee (University of Michigan (US)), Prof. Wenjing Wu (University of Michigan)
- UM vmware cluster lost an MD3800i iscsi storage device(umvmstor03), so images of a few critical services, including afs server, cfengine, glpi, local squid servers were lost permanently, we had to rebuild all those services from scratch and restore the configuration from Amanda backup
- MSU site Received 8x R6525 from September order (Rome 16C/32HT CPUs , 256G RAM).
Racked and running benchmarks (including boinc on/off and now working on hepscore).
HS06 16.44/HT (no boinc) - Did an update on one of the ATLAS gatekeepers (condor 9.0.8->9.0.9, condor-ce 5.1.2->5.1.3), all existing jobs got lost on this gatekeeper (jobs in both condor and condor-ce sat in idle status for over an hour, had to remove all the jobs, and let new jobs come in.) For the other gatekeeper, we decided to drain it first before doing the update.
- For BOINC backfilling jobs, we changed the cgroup and cpu percentage used by BOINC jobs, it shows the impacts on grid jobs’ CPU efficiency drops by 6%, but the overall CPU utilization also dropped by 3%.
- Enabled WebDAV doors on all our existing dCache doors and changed CRIC to use them via the webdav.aglt2.org RR entry. Also configured the dCache doors to prefer IPV6 instead of IPV4 via http/webdav transfers.
-
13:00
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))
UC
- Phase two move this week. In the process of getting workers and storage back online
- Upgraded condor on the workers to 9.x
- Working on bringing up the new UC PerfSONAR nodes
- Slowness on uct2-s28. Applied a couple of tunings that appeared to have helped. Applied the tunings to our other MD3460 nodes as well
- Applying the configuration to prefer IPv6 to the dCache pools
IU
- Nothing to report
UIUC
- Upgrading condor on the workers to 9.x
-
13:05
NET2 5mSpeaker: Prof. Saul Youssef
-
13:10
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
UTA_SWT2:
We had a problem with the limited bandwidth into the cluster and the job mix since our fair-share allowed for some reconstruction jobs to try to run. We have set the Panda Queue to only run Evgen and Simul jobs and things have improved.
SWT2_CPB:
We closed the ticket related to transfers from RAL to SWT2_CPB. Partly this was caused by the original configuration issues with the WebDAV service and the a desire by ddm-ops to seemingly move a large amount of data. Hiro configured a link in FTS to provide enough priority to clear the backlog of transfers.
OU:
Site running well. We upgraded xrootd on se1 proxy gateway to 5.4.1 RC1, and so far, that's running well and hasn't hung or crashed.
- The last couple of weeks have been a good running period with fairly steady production.
-
13:15
→
13:20
WBS 2.3.3 HPC Operations 5mSpeakers: Lincoln Bryant (University of Chicago (US)), Rui Wang (Argonne National Laboratory (US))
TACC
- Down to 15% of allocation remaining, have 6 weeks left.
- Long queue times are slowing down our allocation burn.. a few mitigations in place
- Harvester will now queue for 1 week. Previously it was configured to remove jobs after 48H which was killing us in getting through the queue. TACC prioritizes length-of-time-in-queue over other variables.
- Reduced job size from 100 to 10 nodes. Smaller jobs should get through the queue faster.
NERSC
- Charging policy for NERSC is now as such:
- Cori KNL "charge factor" is 0.2 vs 1.0 for Perlmutter in AY2022.
- However, Perlmutter is still free until some date TBD, "likely in the second half of the year".
- Given the above, we restarted jobs on Cori for the foreseeable future, but we should figure out how best to spend down the allocation once Perlmutter starts charging.
- Running jobs through NERSC_Perlmutter_Test queue currently, hoping to see a success soon.
- Rui and I will figure out optimal job sizing afterwards. Right now configured to use 128 (all) threads per job. Maybe do something similar to TACC and pack 16x 8-thread PanDA jobs per Slurm job.
-
13:20
→
13:35
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
- 13:20
-
13:25
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
13:30
Analysis Facilities - Chicago 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
Today is the 'grand opening' of the uc af. Cecilia Duran (Northern Illinois University) is presenting at today's IB meeting https://indico.cern.ch/event/1120534/.
-
13:35
→
13:55
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind (Brookhaven National Laboratory)
- OSG 3.5 EOL officially pushed back to May 1st.
- dCache IPV4 preference was due to a jvm default setting (tracked down by Hiro, thanks!). Simple fix implemented at BNL, though this is still not RFC compliant and dCache developers are working on a longer term fix.
- FedOps: issues with AGLT2 SLATE squids (root partition usage) - looking into implementation of run status alerts; planning to migrate Drupal support pages to new US ATLAS web site
-
13:35
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
-
13:40
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
We have a number of updates needed for squid devOps, and we need to setup training for a new squad of two, to relieve Lincoln.
XCache
- working fine.
- after a lot of discussions starting deployment of XCache in Birmingham to be used in VP context. depending how that will go we might see more sites in UK going cache only.
VP
- working fine.
- Some more parts of Rucio integration got into production
ServiceX
- changes in CI/CD
- multipath support is ready for production next week. Will improve reliability.
Analytics
- new data streams from GeoModel
- PerfSONAR changes working fine.
- ES move will be completed before end of the week.
- 13:45
-
13:55
→
14:05
AOB 10m
-
12:00
→
12:10