US ATLAS Computing Facility
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Lin (University of Wisconsin), Matyas Selmeci
OSG 3.5.19 / 3.4.53
- XRootD 4.12.3
- HTCondor-CE 4.4.0 (3.5 only): identify misconfigured job routes during startup
- HTCondor 8.9.7 (upcoming)
EL 8 support
- Aiming for full OSG Software Stack support by the end of the year
- osg-wn-client and osg-oasis packages are all available in osg-testing.
- Currently working on verifying XRootD, the osg-wn-client tarball, and Frontier Squid
- Are there US-ATLAS plans for updating to EL 8?
- OU: new EL8 (?) cluster in the fall and will need WN, XRootD storage, and CE packages
- BNL hopes to transition starting next year
-
13:20
→
13:35
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
-
13:20
Tommy Shearer (IRIS-HEP Fellow / UM Undergraduate) 15m
Tommy will present his IIRIS-HEP project to create use-case specific dashboards and interfaces to networking data to support users, admins, network engineers and network researchers.
-
13:20
-
13:35
→
13:40
WBS 2.3.1 Tier1 Center 5mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
-
13:40
→
14:00
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))-
13:40
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)
Painful dcache update:
TLDR: updating to 6.1.5 or 5.2.22 to pick up a TPC fix caused lots of trouble.
Current state: We have now updated our headnodes to CentOS 7 with a patched version of 5.2.22 on all pool nodes.
We will upgrade to the next official 5.x or 6.x release.
We still need to determine why some of our pools nodes are transferring so slowly.
We currently suspect a switch interface dropping packets.
For lots of details see below.We had a switch issue (memory corruption in one switch unit) which caused most of the system to be disconnected from LAN on 20th June because all our vmware machine links are from this switch, The incident lasted 2 hours.
We see occasional disconnections from our 2 gatekeepers to condor work nodes which cause condor killing these jobs, and the disconnection from gatekeepers were caused by the high usage of swap.. We increased the RAM on them, it seems to resolve the issueWe keep improving BOINC with more intelligence and easy control, during downtime, BOINC jobs fill all the job slots.Details on painful dcache update:
We started at 5.2.16 and wanted to enable fixes for TPC which were in 6.1.5 or 5.2.22 released Wednesday 3-Jun-2020.
We upgraded to 6.1.5 on Thursday 4-jun-2020 (basically yum update dcache) and that seemed to go well.
However pools started to complain of possible corruption of the BerkeleyDB meta data and require a restart to work.
We submitted a support request but didn't hear back before the weekend.
We tried to downgrade to 5.2.22 (which still had the TPC fix we needed).
But then the SRM Domain would not start, complaining about missing IGTF_LOA_CEDAR
dCache support said this was caused by running a 6.1.5 gPlazma with a 5.2.x SRM but we were not.We then tried all sorts of steps:
downgrade to original version 5.2.16
dcache uninstall, reboot, reinstall
upgrade back to 5.2.22
clean and vacuum DB
reboot all dcache systems
Only turn on the headnode with the main dcache services and no storage node.
All of this didn't work with SRM not starting (same error about IGTF_LOA_CEDAR).Since our headnodes were the only dCache servers still running SL6,
we decided to rebuild them (in-place) as CentOS7.
This worked and we restored the ZFS pool hosting the DBs on both headnodes
and reinstalled Postgresql-11 and tried to bring dCache up.
The same problem was still present!At this point it had to be something in the DB.
Dmitry at FNAL told us how to turn on debug logging for the SRM domain
and we found out that the last thing it did before dying was read the 'srmuser' table.
The problem was that 6.1.5 related credential stuff was stored in the 'encoded' field
in the srmuser table. We manually deleted all entries refering to IGTF_LOA_CEDAR
Then restarting SRM worked on 5.2.22.However, we then started seeing the other problem again with pool sporacically stopping:
"Pool disabled: Meta data lookup failed and a pool restart is required: (JE 7.3.7) Environment must be closed, caused by: com.sleepycat.je.ThreadInterruptedException: Environment invalid because of previous exception: (JE 7.3.7) /dcache5/pool/meta java.lang.InterruptedException THREAD_INTERRUPTED: InterruptedException may cause incorrect internal state, unable to continue. Environment is invalid and must be closed."With help from Tigran we installed and used btrace (Java trace utility).
We put Tigran's ssh key on one of the pool nodes that had the most problems.
Tigran setup "traps" to get the problem. Turns out the pools issue the above message
when they identify that a pool thread was interrupted because they can't know
if there is some potential problem with the pool state at the point.
If the interrupted thread is handled correctly it doesn't cause an issue.
Turns out the interruptions that triggered the problem were transfers timing out.Tigran provided us with a patched 5.2.22 that we put on all pools that were having problems
It has been working find since we deployed it 11-Jun-2020.
IN2P3 seemed to be having the same or a similar issue.
Since then we and IN2P3 tested 2 more tentative releases from Tigran on our most sensitive storage node, one unsuccefull, one successful. -
13:45
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))
Finished upgrading all dCache nodes to SL/CentOS7. Adding 2PB to DATADISK and 100TB to LOCALGROUPDISK
Fixed IPv6 routing issues from UC. Also fixed broken perfSONAR nodes at UIUC
Ongoing rucio mover debugging with Ilija, Mario, and Tomas
GGUS 147492: This doesn't appear to be a site issue, I have closed. Who is the correct contact for these types of tickets?
- 13:50
-
13:55
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
UTA
- Submitted paperwork for new purchases:
- ~ 3PB storage
- 12K HSPEC
- Several new support machines
- Problem this weekend with moving atlas-swt2.org registrar by OIT.
OU:
- Mostly smooth operations, except for
- Transfer failures because of (apparently) too many xrdadler32 processes on the proxy server (se1.oscer.ou.edu), which cause timeouts. Investigating with xrootd and fts experts.
- Submitted paperwork for new purchases:
-
13:40
-
14:00
→
14:05
WBS 2.3.3 HPC Operations 5mSpeaker: Doug Benjamin (Duke University (US))
NERSC is running fine because we switched over to premium quality of service. It costs 2 times per node hour
otherwise we would have no throughput. Will run this way till we get back to Uniform Charge rate line.
work continues at TACC. will now scale up the number of nodes and number of events per job.
-
14:05
→
14:20
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
- 14:05
-
14:10
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
- 14:15
-
14:20
→
14:40
WBS 2.3.5 Continuous OperationsConveners: Ofer Rind, Robert William Gardner Jr (University of Chicago (US))
-
14:20
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
- 14:25
-
14:30
Intelligent Data Delivery R&D (co-w/ WBS 2.4.x) 5mSpeakers: Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Andrew Hanushevsky (Unknown), Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:20
-
14:40
→
14:45
AOB 5m
-
13:00
→
13:10