US ATLAS Computing Facility
Facilities Team Google Drive Folder
Zoom information
Meeting ID: 996 1094 4232
Meeting password: 125
Invite link: https://uchicago.zoom.us/j/99610944232?pwd=ZG1BMG1FcUtvR2c2UnRRU3l3bkRhQT09
-
-
13:00
→
13:10
WBS 2.3 Facility Management News 10mSpeakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
Yesterday we had our third meeting to discuss USATLAS planning for Data Lakes / QoS. Meeting notes at https://docs.google.com/document/d/1h5FNHuSA23k1tdQxRZSFIn3AutEv8V98OtkogWm45Pc/edit while our long term planning document is available at https://docs.google.com/document/d/1PwpQpgD1xLpvqewmLW9xfgWVOOANFJlkCfcons74Mms/edit#heading=h.9v3o1er2wax7
- The meeting had a lot of discussion and the main action item is to describe two possible models, incorporating the monitoring and simulation results we have so far. Currently two models are proposed: 1) Ilija's model where all PRIMARY data is on tape and caches are deployed in front of tape and at sites and 2) Saul's model trying to utilize what commercial clouds have been doing (MinIO, networks, cloud services).
-
13:10
→
13:20
OSG-LHC 10mSpeakers: Brian Lin (University of Wisconsin), Matyas Selmeci
OSG 3.5.27/3.4.57 (this week)
- osg-wn-client installs are failing due to a newly released xrootd4 package in EPEL. New gfal2 build is in osg-testing for 3.4/3.5 that fixes this issue. In the meatime, the following workaround can be used:
yum install --exclude xrootd4 osg-wn-client - HTCondor 8.9.9 (upcoming)
- XRootD 4.12.5 (3.5 only)
Other
- EL8: XRootD and HTCondor are nearly ready; CE packages are still under dev
- osg-software, osg-int, and vdt-discuss mailing lists are going away (replaced by software-discuss and site-announce) Nov 20. Subscribers are not being copied over from the old lists. Expect an email with subscription details this week.
- osg-wn-client installs are failing due to a newly released xrootd4 package in EPEL. New gfal2 build is in osg-testing for 3.4/3.5 that fixes this issue. In the meatime, the following workaround can be used:
-
13:20
→
13:35
Topical ReportsConvener: Robert William Gardner Jr (University of Chicago (US))
-
13:20
TBD 15m
-
13:20
-
13:35
→
13:40
WBS 2.3.1 Tier1 Center 5mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
-
13:40
→
14:00
WBS 2.3.2 Tier2 Centers
Updates on US Tier-2 centers
Convener: Fred Luehring (Indiana University (US))- Smooth running in the last weeks - no outages, few tickets.
- I am trying to close last quarter's capacity numbers. I am chasing the sites one by one to confirm that they have the correct numbers for September 30 to be put on the v53 sheet:
- https://docs.google.com/spreadsheets/d/1nZnL1kE_XCzQ2-PFpVk_8DheUqX2ZjETaUD9ynqlKs4
- Please check the CPU numbers and the Storage numbers (column L).
- Also please check the facilities services information on that sheet: https://docs.google.com/spreadsheets/d/1_fKB6GckfODTzEvOgRJu9sazxICM_RN95y039DZHF7U/
- v53 will be completely locked soon probably later today
- v54 for the quarter ending on December 31 is open
- Lincoln spotted a real issue with the pilot where the pilot itself ended up with 5 GB of memory causing memory exhaustion.
- Paul Nilsson found that a logging service was leaking memory in some cases. Rather than fix the service, it was possible to stop using it. There should be a new pilot version later this week.
- How are we doing on moving to OSG V3.5?
- Need to check the October accounting numbers. September's numbers were ~correct.
- Every site should replace their head nodes / gate keepers / control nodes / etc. that are out of warranty.
- We don't want down time caused by failure of this type of server.
- I will be asking you how old your servers are.
- Having a small meeting with Mayuko, myself, Shawn, Mark, Ofer, and Judith to start to understand stuck transfers.
-
13:40
AGLT2 5mSpeakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)
1 ongoing ticket 149378: a big percentage transfer failure . We found one of the dcache door nodes (dcdum01) having errors about connecting to the head node, we rebooted the node and it fixed the problem.
2 solved ticket:
149268 : files offline during transfer, this is because one dcache pool msufs04_2 became offline, and it was a quick fix after we restarted the pool.
149328: no transfers for several hours. Our dcache head node (hosting pnfs and chimera database) died (raid controller failure) due to room overheating (machines shut themselves down). We had to migrate the system disks and nvme disks (hosting the database data) to another machine, and bring that machine up as the new head node.
Software updates:
We did (both sites) update/catchup the OS+FW on our core EX9208 Juniper routers. The update at MSU had to be partially rolled back because of bug causing instability. After the update of routers, the heat load observed is higher in most blades. Most MSU switch blades returned to normal temperature after partial OS roll back.
We purchased and built 3 new Dell R7525 machines for the UM vmware cluster. We have migrated all vm instances from the old R740 Intel cluster to the new R7525 AMD cluster. We experienced a lot of network issues (most vms have partial network connectivity and require migrating to another host to resolve the problem)during the migration and we are identifying/resolving it gradually.
OS:
In the process of upgrading more SL6 servers to CC7, already finished dns server upgrade.
Hardware:
Replaced one more sick PDU in MSU T2
At MSU T2. Noticed dust buildup on the EX9208 air filter and vacuumed from outside.
At UM T2, we hired a company to clean the dust in the server room, under the rack, outside of the machines.
-
13:45
MWT2 5mSpeakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))
One of the md3460 dCache servers crashed again yesterday evening. Back online as read-only for now while we troubleshoot
Investigating pilots that are using a lot of memory and are OOMing workers
Ongoing 1099 error debugging. We have been able to replicate timeouts in `rucio download` from a worker (iut2-c285)
Continuing to debug poor IU IPv6 performance
Investigating TPC HTTP smoke test issue with GermanGrid CRL
-
13:50
NET2 5mSpeaker: Prof. Saul Youssef (Boston University (US))
Smooth operations except for one black hole worker node and a several hour DDM outage due to a power issue at MGHPCC affecting NoX networking equipment.
BU campus networking has made ipv6 progress...contacting ESNet to coordinate re: LHCONE
Working on: OSG 3.5 upgrade, HTTP-TPC, NET2<->NESE networking
-
13:55
SWT2 5mSpeakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))
-
14:00
→
14:05
WBS 2.3.3 HPC Operations 5mSpeakers: Doug Benjamin (Duke University (US)), lincoln bryant
-
14:05
→
14:20
WBS 2.3.4 Analysis FacilitiesConvener: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:05
Analysis Facilities - BNL 5mSpeaker: William Strecker-Kellogg (Brookhaven National Lab)
-
14:10
Analysis Facilities - SLAC 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:15
Analysis Facilities - Chicago 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
-
14:05
-
14:20
→
14:40
WBS 2.3.5 Continuous OperationsConvener: Ofer Rind
-
14:20
US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5mSpeakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
-
14:25
Service Development & Deployment 5mSpeakers: Ilija Vukotic (University of Chicago (US)), Robert William Gardner Jr (University of Chicago (US))
-
14:20
-
14:40
→
14:45
AOB 5m
-
13:00
→
13:10