US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Lin (University of Wisconsin), Matyas Selmeci
OSG 3.5.34
- Python 3 Gratia probes
- CVMFS 2.8.1 (also in OSG 3.6)
Impending updates
- HTCondor 8.8.13 in OSG 3.5 release
- HTCondor 9.0.0 in OSG 3.5 upcoming and OSG 3.6
- HTCondor-CE 5.1.0 in OSG 3.5 upcoming and OSG 3.6
- XRootD 5.2.0 (or 5.1.2?) in OSG 3.5 upcoming and OSG 3.6
- XRootD, HTCondor, and HTCondor-CE in EL8
HTCondor sites should register for HTCondor week! https://agenda.hep.wisc.edu/event/1579/
-
13:20
→
13:35
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
-
13:20
TBD 15m
-
13:20
-
13:35
→
13:40
WBS 2.3.1 Tier1 Center 5mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
-
13:40
→
14:00
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))- Relatively tough period with various problems - I ask each site about drops in the number of jobs profile.



- Quarterly reporting due!!!
- Shawn wants my report on Friday so please enter your information by the end of Thursday.
- Update the capacity sheet (https://bit.ly/usatlas-capacity) as of 3/31 with purchases and retirements.
- Update the services sheet (https://tinyurl.com/yytakxdr) as of 3/31 with any new versions of middleware, storage system software etc.
- Still not able to implement http-tpc at xrootd sites. Still asking all sites to jump to OSG 3.5
- Need to check the March accounting numbers. Wenjing has agreed to help. After a revision ~1 week ago the number are looking much more reasonable.
- Big discussion at the Monday ADC Technical Coordination meeting about IAM / Tokens. ADC was not planning to switch to tokens before the end of Run 3 but it seems Globus has forced their hand.
- What about IPV6?
-
13:40
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)
1). working on provisioning with the new switches.
2). GGUS ticket about dCache transfer error caused by fiber break of merit network starting from 9am 12nd Apri, the provider is still working on the repair.
3) Had over 1000 production jobs overusing the memory, either killed by pilot or system oom killing.
4) had high memory jobs (>6G) flood (>4000 cores), we put a 1500 core limit on the CRIC PanDA queue, and also modified the gatekeeper routing rules, to make sure high memory jobs only go to the big memory work nodes.
5) rebuild most of the work nodes with dedicated partition for BOINC jobs (which originally use /tmp as work directory) to avoid interference with system and condor jobs
6) received 4 recently purchased lenovo/forsa work nodes, working on provision
-
13:45
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))
-
13:50
NET2 5mSpeaker: Prof. Saul Youssef (Boston University (US))
1. Rolling upgrade to centos 7.9 almost done.
2. Smooth operations otherwise modulo one dip in production seemingly for just lack of panda jobs.
3. Preparing to purchase 2-4 gateway nodes for NESE Ceph storage.
4. NESE core networking will be upgraded on April 22. NESE_DATADISK,_SCRATCHDISK will be down.
5. We will have a 5 minute network outage tomorrow after 5 to set up ipv6 BGP peering between BU and ESNet.
6. We have a container with Xrootd 5.1.1, successfully exporting GPFS. Testing now, plan to jump to 5.2(?) 5.1.2(?) when it appears in OSG.
-
13:55
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
- Relatively tough period with various problems - I ask each site about drops in the number of jobs profile.
-
14:00
→
14:05
WBS 2.3.3 HPC Operations 5mSpeakers: Doug Benjamin (Duke University (US)), lincoln bryant
Very good running at NERSC. Increased number of running workers from 5 to 8.
Each worker (batch submission) has 200 KNL nodes. 6 hours per submission.
Lincoln is running test jobs on his test instance to make sure he can operate the production instance using his credentials (grid and globus).
No running at TACC - Rob submitted revised proposal for more time at TACC.
-
14:05
→
14:20
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
- 14:05
-
14:10
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:15
Analysis Facilities - Chicago 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
-
14:20
→
14:40
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind
- Changing protocol from SRMV2 to gridftp in OSG topology seemed to fix downtime declaration problem for SWT2
- Reviewing site topologies to identify remaining SRMV2 protocol settings to change
- Site admins please notify Mark of any downtime issues encountered relating to CRIC, OSG topology, Hammercloud, Switcher, etc....we are trying to compile and document the problems
- BNL now fully webdav for WAN transfers
- See attachment for list of remaining sites still on SRM/gsiftp (Alessandra)
- Need a plan for Tier-3s - xrootd seems viable, little use of http
- SWT2 and BU site plans to move to OSG 3.5?
- SWT2 next week?
- OU as soon as XRootd 5.1.2 (or 5.2) released and stable; gatekeeper will be updated when new hardware arrives, expected in "a month or two"
- BU in process, testing 5.1.1
- Frontier-Squid DevOps meeting today
- Enabling k8s writing of logs to standard out caused pod flapping at MWT2 - need to turn off for stable running; continuing to investigate cache hit rate, will know more next week
- Quarterly reports
-
14:20
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
- 14:25
- Changing protocol from SRMV2 to gridftp in OSG topology seemed to fix downtime declaration problem for SWT2
-
14:40
→
14:45
AOB 5m
-
13:00
→
13:10