US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Lin (University of Wisconsin), Matyas Selmeci
OSG 3.6 (next week)
- HTCondor 8.9.12 delayed, releasing 8.9.11 instead
- XRootD 5.1.0 not yet ready, excluded from initial release along with XCache and other packages dependent on XRootD
- HTCondor-CE 5.0.0 will make the initial release (https://github.com/htcondor/htcondor-ce/releases/tag/v5.0.0) and we're aiming for new job router config syntax in 5.1.0, with the old job router syntax continuing to be the default
- Otherwise the announcement from earlier this month still stands [1]
OSG 3.5.32 (aiming for next week)
- HTCondor-CE 4.5.0 (https://github.com/htcondor/htcondor-ce/releases/tag/v4.5.0)
- gratia-probe: added Python 3 support
- osg-ce: Fix some HTCondor-CE dependencies in preparation for new HTCondor-CE versions
- osg-scitokens-mapfile: initial packaging including HTCondor-CE mapfile for VOs with SciTokens issuers (requires some manual configuration)
[1]
In February 2021, we plan to start a new release series for the OSG Software Stack. The start of a new release series is when we introduce major changes to the OSG Software Stack. It also give us and opportunity to drop packages that are no longer useful to the community. We are also expecting the following major software updates: - Dropping GridFTP, GSI, and Hadoop packages - Adding packages to support token-based authentication - HTCondor 8.9.12 followed shortly by 9.0.0 - XRootD 5.1.0 - HTCondor-CE 6.0.0: new default job router syntax - Gratia probe - consolidating all batch system probes into the HTCondor-CE probe - dropping storage probes We are planning a major change to the way that we release software. In OSG 3.5, we have a rolling release repository for early access to tested packages. In OSG 3.6, rolling releases will become the standard way of updating packages for the OSG. We will still make announcements when there are security issues or critical bug fixes. See https://github.com/opensciencegrid/technology/blob/579559466189578d772be531222c07303a675de8/docs/policy/software-release.md for a draft of the OSG 3.6 Software Release Policy. Packages will not be promoted to the release repository the day before a weekend or holiday, unless there are extenuating circumstances. We hope to make this process as easy as applying updates to your laptop. In addition, we have updated our Container Release Policy to more closely match the software in our RPM repositories. For more details, consult our draft policy: https://github.com/opensciencegrid/technology/blob/227362f60fa119e5b468c7ccb64496d7daf84e10/docs/policy/container-release.md
-
13:20
→
13:35
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
-
13:20
SSL Year 3 Plans Recap 15m
Quick recap of talk given at yesterday's IRIS-HEP steering board meeting
Speaker: Robert William Gardner Jr (University of Chicago (US))
-
13:20
-
13:35
→
13:40
WBS 2.3.1 Tier1 Center 5mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
- smooth operation in general;
- GGUS ticket (https://ggus.eu/index.php?mode=ticket_info&ticket_id=150548), a large number of xrootd requests on dCache, exceeding limit, caused some transfers to fail. It's fixed by increasing the max number of movers for xrootd (and http and NFS movers as well) to 50K
-
13:40
→
14:00
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))- Stable running except CERN network outage last Friday.



- Still trying to grok out how to update the average CPU power for a site in the CRIC era. If someone knows please contact me.
- Wenjing tried update the standard value in the OSG topology: APELNormalFactor
- Brian thinks that this is still the place and certainly for GRACC it must be but when Wenjing changed this value it did not change in CRIC. Manual changes to the CPU power by Wenjing we reverted overnight;
- According to Alexey Anisenkov the value being used is found at: https://atlas-cric.cern.ch/atlas/pandaqueue/list/?atlas_site=AGLT2&state=ACTIVE
- Change AGLT2 in the above URL to look at other sites.
- Alexey told us that there are 4 places where the CPU power can be set and these places override each other in a predetermined order.
- Please update the info about the CPU and storage on the new v55 tabs of the capacity sheet:
- Enter changes made starting Jan 1.
-
13:40
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)
Tickets and operation:
- 150558 AGLT2 failing pilots : solved : one worker node cvmfs needed restart
- pilot error 1151: "File transfer timed out during stage-in"
Fred notified us of a surge of such errors at AGLT2 late last week.
A random sample of jobs were checked and all pointed at one dcache server (msufs04)
One of the 4x10G ports was showing in-errors.
Taking that port down reduced the number of errors, but still low rate.
Another random sample looked at yesterday were still from msufs04.
We have some packet discards, at times, and it seems correlated to load spikes.
The 3 similar nodes msufs02,03,04 all have some in-discards, at times.
msufs04 has ~5x more (maybe because it has more hot files recently, from period as rdonly).
These nodes have exhausted their swap file.
CPU Load and memory pressure may be what is causing the remaining in-discard issue
But, it does not look like in-discards or swap usage is anything new.
So this may be a fact of life that we have a low rate of these failures.
Just set msufs04 readonly for a few hours and will restart dcache (today) to see if that helps
and even reboot(tomorrow while on site).- ddm support notified Fred/us that CNAF tape to AGLT2 fails 100%
Doesn't seem to be a problem at AGLT2, forwarded to fts experts.Hardware:
- UM site is working on replacing more PDUs and getting fan door replacements.
(more PDUs start to die on the compute side, and more door fans are dying in the Tier2 room)- rebuilt 2 more R710 nodes (with 108TB disk each) as slate nodes, to serve as extra xcache nodes.
- rebuilding another node from SL6 to CentOS7: GLPI ticketing and web server for muon calibration
-
13:45
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))
UC
- Observing an occasional increase in network usage. We also see an increase in job failures during the times when our egress traffic is maxed out
IU
- IU GigaPoP maintenance caused a spike of job failures Feb 5
UIUC
- Water chiller failure over the weekend of Feb 6-7 caused the UIUC site to be offline for emergency maintenance
- Retired 2224 cores (mwt2-c[001..066])
- New workers in the process of being built
-
13:50
NET2 5mSpeaker: Prof. Saul Youssef (Boston University (US))
SLATE squid registered in AGIS, getting to jobs at NET2 in FRONTIER_SERVER
BU Networking testing their ipv6 infrastructure
Few jobs wedging local file systems, causing jobs to be killed after 96 hours by SGE
Ongoing:
OSG 3.5 prep
Xrootd 5.0.3 testing for replacing gridftp
Preparing to buy worker nodes
- 13:55
- Stable running except CERN network outage last Friday.
-
14:00
→
14:05
WBS 2.3.3 HPC Operations 5mSpeakers: Doug Benjamin (Duke University (US)), lincoln bryant
Tried to run NERSC with low QOS to save allocation hours (lower burn rate). few jobs ran so switched back to regular QOS.
NERSC now in monthly maintenance downtime.
ALCF - changed to run 1024 node submissions instead of 384 node submissions to see if ATLAS event throughput increases. - large node submissions schedule faster. Of course we are now behind many very large jobs. with 384 jobs it typically takes 5-6 days for a 384 node to start.
TACC - various site issue have prevent ATLAS from running.
ATLAS DPA (Rod Walker) increased the number NQUEUELIMIT_opportunistic=10k to 15k. TACC had sucked up all the jobs (even though every US HPC has dedicated tasks) and ALCF could not ramp up.
Now all HPC's have lots of activated jobs - https://bigpanda.cern.ch/jobs/?workqueue_id=1002
Doug is joining BNL SDCC 2-March-21 and will be transitioning out of HPC operations.
-
14:05
→
14:20
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
- 14:05
-
14:10
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
- 14:15
-
14:20
→
14:40
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind
-
14:20
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
-
14:25
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
all XCaches moved to 5.1.0rc6 All stable.
new SLATE chart that supports multinode deployments.
AGLT2 added two more XCache servers.
VP
* support for prefix delivery to Rucio.
* now has token authentication for all endpoints changing the system.
In order to actually use multinode caches, I have to change Rucio so it gets cache prefixes not from AGIS/CRIC but from VP.
-
14:20
-
14:40
→
14:45
AOB 5m
-
13:00
→
13:10