US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:15
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
- 13:00
-
13:15
→
13:25
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
First item
We should discuss our plans for Data-Lake prototyping and network challenges.
- We have a mix of dCache and Xrootd across our facility and we need to think about how best to use our resources to prototype for data lake exploration
- There are discussions underway in IRIS-HEP and WLCG DOMA about network challenges and demonstrators. How will our facilities contribute and participate?
We can discuss briefly on these items and related topics, but we will likely need to organize a separate meeting to plan our efforts.
Second item
As we have discussed in our facilities meetings, it would be good to have a focused meeting to discuss our upcoming network procurements, utilizing the additional critical infrastructure funds we received in this funding round.
- Important characteristics: Deep buffers, support for VXLAN, programmability, base chipset, etc
It is important to discuss our purchase plans in the context of switch/router features we should look into having, especially considering the prototyping, challenges and planning we are undertaking to prepare for HL-LHC.
For both items we need to identify who is interested and find a meeting time (via Doodle?)
Shawn
-
13:25
→
13:35
OSG-LHC 10mSpeakers: Brian Lin (University of Wisconsin), Matyas Selmeci
OSG 3.5.25 and 3.4.55
- Singularity 3.6.3 (minor security fix, 3.4 only)
- Blahp 1.18.48 https://github.com/htcondor/BLAH/releases/tag/v1.18.48
Miscellaneous
- CEs that need to be updated to HTCondor-CE 4 (available in OSG 3.5). Documentation available for uppdating to 3.5 here: https://opensciencegrid.org/docs/release/release_series/#updating-to-osg-35
- atlas-ce.bu.edu
- gk01.atlas-swt2.org
- gk04.swt2.uta.edu
- grid1.oscer.ou.edu
- osgitb1.nhn.ou.edu
- tier2-01.ochep.ou.edu
- OSG 3.6 teaser announcement coming soon! Highlights include:
- Deprecating GSI/GridFTP in favor of token-auth and XRootD ()
- Moving to a rolling release
- Considering changing our container tag process
- We're interested in deprecating unused OSG environment variables (https://opensciencegrid.atlassian.net/browse/SOFTWARE-3011). Does ATLAS use any of the following?
- OSG_APP
- OSG_DATA
- OSG_DEFAULT_SE
- OSG_HOSTNAME
- OSG_LOCATION
- OSG_STORAGE_ELEMENT
-
13:35
→
13:40
WBS 2.3.1 Tier1 Center 5mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
-
13:40
→
14:00
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))- Paperwork for the next funding should be prepared and returned to Stony Brook.
- Relatively good running this week
- BUT more tickets than we have had recently
- Please keep the site information Google sheet with information up to date as you make changes to your site's configuration. Thanks go to Ofer and Mark for organizing this.
- https://docs.google.com/spreadsheets/d/1_fKB6GckfODTzEvOgRJu9sazxICM_RN95y039DZHF7U
- Don't forget that quarterly reports will due (today is the last day of the current quarter).
-
13:40
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)
Software
upgraded our dCache from 5.2.25 to 6.2.5
- mostly successful with some hurdles.
- The services were down only for 1-2 hour as we worked through these hurdles.
- There is a dependency on zookeeper as "the ZooKeeper ensemble which is used by dCache
is required to be of version 3.5. Version 3.4 will no longer be supported."
But zookeeper itself nor the dependency is included in the dcache rpm
and has to fetch it on the apache sites.
- dcache now uses systemd as .e.g. the dcache start command redirects to systemctl
- There is a remaining concern that rebooting the dcache doors did not restore the services.
It seems like 'dcache start' does not succeed for a while, still not clear.
- One remaining issue related to systemd is that we no longer have log files for each dcache pool
and thus no longer send these records to ELK.
We may revert the dcache config to use the init method.
We were told "DESY uses journalbeat to ship logs to elastic search" which would also be interesting.
- from Tigran: We have started a 6.2 upgrade guide
https://www.github.com/dCache/upgrade-guide-62/tree/master/UPGRADE62.md
In addition we plan in November a dedicated systemd related sessionTickets (none currently)
- 148666 about file transfer efficiency.
found to be one dcache pool node with a slow and painful failure of one of the system disks in RAID1
Disk replaced with no further issue.Site issue
- MSU site fire alarm 16-Sep-2020, power off, including UPS, fire trucks.
VESDA system detected smoke, most likely from neighboring dept server room sharing cooling.
Philippe happened to be on site to fix for above ticket, quick recovery of core services.
Lost a few of the older disks from RAID6 arrays.
A number of aging PDUs died or no longer could control their outlets.
5 PDUs replaced next day to complete recovery of worker nodes. -
13:45
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))
UC
- New compute racked and cabled. Working on configuring the new switch
- SLATE cluster rebuilt last week due to kubernetes issues
- Storage controller went offline last night ~11pm. Back online now
IU
- All new compute benchmarked and online
- IPv6 issues have been isolated to the Brocade switch. Reverting IPv6 on the workers until this can be resolved with the vendor
UIUC
- Discussing new compute purchase
-
13:50
NET2 5mSpeaker: Prof. Saul Youssef (Boston University (US))
New workers are up and running.
NESE_SCRATCHDISK added at ADC request.
Low level Squid stress issue and GGUS ticket. Still need to fix this. Ready for a SLATE squid?
Replacement memory dimm in SLATE node presumably will stop occasional crashes.
Dying switches caused ~50% failure rate on some nodes. Replacement switch at MGHPCC. Most affected nodes off in the mean time.
Smooth operations otherwise. -
13:55
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
UTA:
- SLATE node now operational
- Many "Thanks" to Ilija and Lincoln
- A couple of tickets raised due to XrootdFS issues on the GridFTP hosts
- Working on a new version of the SRR reporter for Xrootd
- New compute nodes are being setup / tested / benchmarked
OU:
- Nothing to report, all running well.
- HC failures caused by incorrect RAM request of jobs, being worked on by various people.
- SLATE node now operational
-
14:00
→
14:05
WBS 2.3.3 HPC Operations 5mSpeakers: Doug Benjamin (Duke University (US)), lincoln bryant
Work occurred that week to make the pilot and event service work with python 3. The event service part of the code was not ever tested for python 3 like the rest of the pilot.
Pilot wrapper needs to change for cvmfs free sites and python 3.
TACC is ready for traditional simulation jobs at the level of 500 concurrent jobs. These jobs are less efficient on TACC than traditional grid nodes because at TACC we use the entire node and there are two significant serial phases that cause 55 out of 56 cores to idle. With Event service there will be only 1 serial phase to the work. Merge (single core activity) can be done elsewhere.
- Question do we want to wait for Raythena to be certified and also to be made work at TACC or do we want to start getting traditional simulation jobs sent to TACC?
- NERSC scratch disk has had problems since last Thursday. Cori is down since weekend because of these problems. Today will get an estimate when the machine will be back
- Started to get Harvester revived at ALCF Theta. testing with some simple tasks.
-
14:05
→
14:20
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:05
Analysis Facilities - BNL 5mSpeaker: William Strecker-Kellogg (Brookhaven National Lab)
-
14:10
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:15
Analysis Facilities - Chicago 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
-
14:05
-
14:20
→
14:40
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind
-
14:20
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
-
14:25
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
SLATE + XCache + VP:
* SWT2 - SLATE OK, XCache OK, VP jobs running very well
* MWT2 - SLATE OK, XCache OK, VP jobs running well, most issues related to our dCache and FZK_LCG2
* AGLT2 - SLATE OK, XCache OK, VP jobs run well
* BNL - XCache manually deployed by Ofer, Xin, VP jobs run well: now in partly "diskless" mode - uses AGLT2_SCRATCHDISK. Will move to fully diskless soon.
* NET2 - waiting on ...
-
14:20
-
14:40
→
14:45
AOB 5m
-
13:00
→
13:15