US ATLAS Computing Facility (CANCELLED)
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Lin (University of Wisconsin), Matyas Selmeci
-
13:20
→
13:35
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
-
13:20
TBD 15m
-
13:20
-
13:35
→
13:40
WBS 2.3.1 Tier1 Center 5mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
-
13:40
→
14:00
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))-
13:40
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)
Update
1) both UM and MSU sites updated both core switches to latest JunOS v18. reminder: During the previous update the MSU router had to be downgraded to v17 because of a memory leak bug causing restarts every ~6h.
Incidents:
1) Some BOINC jobs caused /var/log/message flood with the squashfs error. It happened on ~10 work nodes. Also because of BOINC using squashfs, we are trying to rebuild the WN with bigger /tmp area. (1GB/core)
2) MSU router: we added anti-spoofing filter on the packet source address to protect against potential attacks on our DNS or NTP servers. This filter is applied to the public VLAN. At first, the filter specified only the corresponding public subnet. A sampling ping test did not spot any problem. But a few hours later, several hundred condor jobs lost heartbeat and were dropped. It was then found that a fraction of the pings were failing from UM to MSU (but not from MSU to UM?!) on the private subnets. The solution was to add the private subnet to the filter definition. In fact UM had to do a similar thing earlier. It is still not clear why this is needed. This is still being investigated before moving to the new routers and campus research networks.
3) Tried to update from OSG3.5 to OSG3.6 which includes the update in both Condor and Condor-CE.
With Condor, we hit authentication problem after updating the head node from 8.8.12 to 8.9.11 (The work nodes were updated to 8.9.11 a month ago). We could not resolve the authentication issue after trying different configurations, so we rolled back the Condor head node to 8.8.12.
With Condor-CE, after the update, Condor-CE could not receive any new jobs due to authentication failure. After many times of reconfiguration, we still could not resolve the issue, so we rolled back the update, but the authentication issue persisted with the ATLAS jobs (It works with OSG jobs). It is possibly caused by a bug in Condor-CE. (ATLAS jobs keep using an old security session, and Condor-CE rejected it, and send back a rejection message, but this message is blocked by the CERN firewall, so ATLAS jobs could not get this rejection message). Condor-CE started to receive new jobs after ~12 hours (the security session expired at the ATLAS jobs side).
4) update on "pilot error 1151" problem: finished resolving the fewer remaining errors after rebooting this dcache pool node (msufs04). It seemed to have some ~5k hanging processes which were likely locking some memory resources.
-
13:45
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))
-
13:50
NET2 5mSpeaker: Prof. Saul Youssef (Boston University (US))
-
13:55
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
-
13:40
-
14:00
→
14:05
WBS 2.3.3 HPC Operations 5mSpeakers: Doug Benjamin (Duke University (US)), lincoln bryant
-
14:05
→
14:20
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:05
Analysis Facilities - BNL 5mSpeaker: William Strecker-Kellogg (Brookhaven National Lab)
-
14:10
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:15
Analysis Facilities - Chicago 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
-
14:05
-
14:20
→
14:40
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind
-
14:20
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
-
14:25
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
-
14:20
-
14:40
→
14:45
AOB 5m
-
13:00
→
13:10