US ATLAS Computing Integration and Operations
-
-
13:00
→
13:15
Top of the Meeting 15mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Robert William Gardner Jr (University of Chicago (US))
Goal today is to attempt to standardize on configs for storage and worker nodes.
Compute - Summary from Shawn
- 6.6k as configured
- based on R630, 56 slots per server
- mobo has 2x10g, 2x1g
- IOPs - went with 2x480G SSDs + 1 500G NLSAS
- Most everyone should have the original quote
- Concerns or questions?
- Saul: worried about local disk; Shawn: 20G / job slot, perhaps would need more.
- What is the ATLAS requirement? 30G/job?
- As for more disk
- R0 over 4 disks? (higher failure rate, but its a worker node)
- Memory configuration (for the 192G). Each CPU has 2x32G and 2x16G. What about 256G option? 8x32Gs.
- Agreement on this config with 192G and 4x480G SSDs
(see info in the chat window)
Storage - Summary from Shawn
- MD3060e 6 Gbps, does not yet support 12 Gbps
- MD3460 does, but it adds $3.3k to the cost of each shelf
- Alternate is to put two H830 in the server, adds negligible cost
- Head node will need to be R730xd (2U server)
- At least 256G of RAM, due to the amount of storage behind it
-
13:25
→
13:35
Capacity News: Procurements & Retirements 10m
-
13:35
→
13:45
Production 10mSpeaker: Mark Sosebee (University of Texas at Arlington (US))
1) Sites are generally not full with production jobs. This is a known issue - not enough jobs/tasks coming into the system. See the MC production talk at the ADC Weekly on 7/5:
https://indico.cern.ch/event/531083/contributions/2164542/attachments/1303757/1947493/MCProd_ADC_2016July05.pdf
2) Otherwise US cloud functioning well - few incidents raised by shifters.
-
13:45
→
13:50
Data Management 5mSpeaker: Armen Vartapetian (University of Texas at Arlington (US))
-
13:50
→
13:55
Data transfers 5mSpeaker: Hironori Ito (Brookhaven National Laboratory (US))
-
13:55
→
14:00
Networks 5mSpeaker: Dr Shawn McKee (University of Michigan ATLAS Group)
-
14:00
→
14:05
FAX and Xrootd Caching 5mSpeakers: Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Andrew Hanushevsky, Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:25
→
15:25
Site Reports
-
14:25
BNL 5mSpeaker: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR))
-
14:30
AGLT2 5mSpeakers: Robert Ball (University of Michigan (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
A problematic NIC (or cable or switchport, not yet localized) was causing intermittent 20 minutes outages on one of our VMWare servers, that just happened to have our primary gatekeeper running on it. This NIC was disabled until we have the time to track down the real cause of the problem, and no further VMWare interruptions have since resulted.
In preparation for storage upgrades at UM and MSU, one dCache pool server at each site is being de-comissioned. One is down already, msufs05, and the second, umfs07, is now 70% drained. Replacement storage at MSU from base Tier-2 funds is now on hand, and will be brought online as soon as it can be prepped.
-
14:35
MWT2 5mSpeakers: David Lesny (Univ. Illinois at Urbana-Champaign (US)), Lincoln Bryant (University of Chicago (US))
Site is running well
APF problem with iut2-gk gatekeeper
- APF not sending enough pilots to iut2-gk for analy queue
- Jose found some "ghost" pilots
- After cleanout, pilots resumed
- Most likely caused by all the networking issues at IU and UC in June
Network problem with 100Gb link at UChicago site on evening of June 25 took MWT2 offline
- I2 reported a monitored instability problem with the 100Gb link at UChicago
- 50% of packets off site were being droppped
- Internally MWT2 did not have any network problems
- SE marked as down; all Panda Qs were offline
- Network admins at UChicago restarted Brocade 100Gb interface on morning of June 26
- Link became stable, no droppped packets
- SE marked online; all Panda Qs came online
- UChicago network admins will work with Brocade as this is the second occurance
Object Store
- Setup on Ceph system
- Stress testing is planned by Doug
- Lincoln has provide the keys and other information
- Still waiting on Doug to do the testing
- Lincoln working with Alex to tune for better performance
FY16 Purchases
- UChicago
- On Hold
- Indiana
- On Hold
- Illinois - Hardware ordered last week (note - nodes must conform to Illinois Campus Cluster configuration and pricing)
- Dell confirmed the order
- Nodes should be delivered and online by end of the month
- 6 x C6320 chassis (4 nodes per chassis, 24 two socket nodes)
- Xeon E5-2680 v4 (56 logical cores/node with HT enabled)
- 11.57 HS06/LC
- 1344 Logical Cores
- Adds 15,550 HS06
- 256GB (4.5GB/LC)
- 10Gb to each node
- Network switches ordered (Juniper QFX5200)
- Racks and installation will be delayed until fall
- Electricians are no available due to summer projects on campus
-
14:40
NET2 5mSpeaker: Prof. Saul Youssef (Boston University (US))
- 14:45
-
14:50
SWT2-UTA 5mSpeaker: Patrick Mcguigan (University of Texas at Arlington (US))
-
14:55
WT2 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:25
-
15:25
→
15:30
AOB 5m
-
13:00
→
13:15