US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
- Among many other interesting presentations at last week's ATLAS S&C, a good summary of ADC activities by D&D: https://indico.cern.ch/event/956962/contributions/4021727/attachments/2120307/3568329/201005_ADCSummary.pdf
- There is an IRIS-HEP sponsored "blueprint" meeting coming up, Oct 26-27: https://indico.cern.ch/event/960587/. Please register and plan on attending if you can.
- Reminder to fill out the facility capacity spreadsheet, https://bit.ly/usatlas-capacity
- Meeting today to discuss Data Lakes and QoS activites for our US ATLAS Facilities. 3 PM Central, 4 PM Eastern. https://umich.zoom.us/j/91315543645 Passcode: 065589
- Meeting tomorrow to discuss Network procurements for the facility. https://umich.zoom.us/j/93616604298 Passcode: 332996
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Lin (University of Wisconsin), Matyas Selmeci
OSG 3.5.26 (not this week)
- CVMFS 2.7.5
- XRootD 4.12.5? RC1 is available in osg-testing, XRootD and OSG devs would appreciate any testing!
Miscellaneous
- OSG 3.6 planning: https://opensciencegrid.atlassian.net/browse/SOFTWARE-4282
- Deprecating OSG env vars: https://opensciencegrid.atlassian.net/browse/SOFTWARE-3011
- Eliminating batch-system specific Gratia probes in favor of a consolidated HTCondor-CE probe
- Dropping GridCF packages
- Token-only authentication, no X.509!
- Sets the OSG 3.5 retirement date to January 2022
- ATLAS XCache images diverging again
-
13:20
→
13:35
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
-
13:20
No topical presentation today 15m
-
13:20
-
13:35
→
13:40
WBS 2.3.1 Tier1 Center 5mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
-
Smooth operation in general, except for a quick drop of running jobs last Wed, due to a PanDA bug.
-
core server migration of the HPSS tape system last Tuesday and Wed. went well, the frontend dCache was still up, impact on production is minimum
-
VP test ongoing, a couple of issues : permission issues on some dCache directories, under investigation; stage-in failures from BNLLake, to be excluded for now till further development from PanDA/Rucio
-
Users can't get files from BNLHPC_SCRATCHDISK -- datasets placed there by mistake in DDM, will be corrected.
-
-
13:40
→
14:00
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))- Quiet week - no major outages on our side
- Change to Panda put in during software week drained a large number of ATLAS sites world wide.
- Still getting few GGUS tickets
- Don't forget your reporting - please provide your reports a couple of days before the drop dead date so that the managers can see it before filing their reports.
- Could everyone summarize their plans to spend the infrastructure/networking money.
- I do understand we are meeting on this tomorrow but I would like to get a feeling of what the plans are.
-
13:40
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)
Incidents:
- One dcache pool went down due to controller memory corruption, rebooting and xfs_repair fixed the problem
- lost power in the entire T2 room on 2nd Oct due to overheating in the room and overloading to the UPS. It took an entire day to resotre power and all services (switch and vmware services took long to recover). We fixed a failed power unit for the UPS, and retired 3 M610/M620 blade servers to save power usage in the Tier2 room. (ggus ticket 148900)
- lost 2 dcache pools (100TB space) due to aging hardware (3 years out of warranty)
- The major 100G switch connecting Tier2 and Tier3 crashed, rebooting fixed it
- ticket 148947: missing dump files for ATLASCALIBDISK. The scripts/cronjob/rpms generating and uploading the dump files were lost during our rebuilding of the dcache head nodes. It is restored
-
13:45
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))
UC
- New UC workers built and benchmarked. Should be online by the end of the week
- HTTP-TPC configured as primary site protocol
IU
- Brocade debugging ongoing
- Hardware issues with the IU SLATE node. Fred is working with Dell to resolve
UIUC
- Converging on a worker configuration for the current purchase cycle
-
13:50
NET2 5mSpeaker: Prof. Saul Youssef (Boston University (US))
Xcache up and running on our NET2 slate node with help from Lincoln. A corresponding ANALY_NET2_VP set up by Ilija and working fine.
Annual MGHPCC down day approaching on October 19. All power will be out for NET2 and NESE.
Missing rucio file noticed in DAST list. Apparently moved "to a WebDAV" enabled site?
Lots of planning happening for NESE tape.
Still a low level semi-issue with a bit more squid failovers than is comfortable (GGUS ticket).
Smooth operation and full site (other than the PanDA dip that affected everyone) otherwise.
-
13:55
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
UTA:
- Jobs running smoothly
- Supporting ANALY-FRESNO-EKS-REMOTE with SWT2_CPB as data source/sink
- Working on deploying recent storage purchases and retiring oldest storage
OU:
- Nothing to report, running well
- HC failures at OU and BNL are caused by jobs not requesting correct RAM -- well, not requesting ANY RAM -- and therefore the pilot kills them because they are running over. Needs to be fixed on HC side.
- Quiet week - no major outages on our side
-
14:00
→
14:05
WBS 2.3.3 HPC Operations 5mSpeakers: Doug Benjamin (Duke University (US)), lincoln bryant
TACC - Lincoln is on vacation this week. Prior to his vacation - Lincoln and Doug started to work splitting up TACC compute nodes to have 7 pilots per node (8 - cores per pilot).
NERSC - Cori is finally back but they still do not know the root cause of the FileSystem problems. Running some small test jobs before starting validation jobs.
ALCF - started Test jobs on Theta. queue depth very deep jobs wait many days till they start. (almost a week) this could be an issue.
-
14:05
→
14:20
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:05
Analysis Facilities - BNL 5mSpeaker: William Strecker-Kellogg (Brookhaven National Lab)
-
14:10
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
- 14:15
-
14:05
-
14:20
→
14:40
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind
-
14:20
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
-
14:25
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
- We are organizing a multi-site k8s federation platform for scalable analysis systems. We held a first meeting yesterday with IRIS-HEP and CMS. Looks like Admiralty can provide a lightweight federation layer. Plan is to connect up UC-River-dev, PRP, UW-Tiger, UNL-Tier2. Will update here as things materialize. URL: https://indico.cern.ch/event/965376/. If you're interesting on participating, please send Ilija/Rob an email - all are welcome!
- All US sites have working SLATE and XCACHE deployed. All sites have a VP queue, all accepting and finishing jobs. Caches will need ~ 1 month to warm up. BNL VP queue now gets payload files from AGLT_SCRATCHDISK. Getting it to upload results requires creating a completely new ATLAS site due to a limitation in Panda (all PQ at a site have to have the same write_lan:0 DDM endpoint).
- Federated Ops
- In context of SLATE, we have a set of security policy documents now that have been reviewed by OSG Security, US CMS Security, and US ATLAS Security (Shigeki): https://slateci.io/docs/security-and-policies/index.html
- These might be useful for local policy discussions at your facility
-
14:20
-
14:40
→
14:45
AOB 5m
-
13:00
→
13:10