US ATLAS Computing Integration and Operations
-
-
13:00
→
13:15
Top of the Meeting 15mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Robert William Gardner Jr (University of Chicago (US))
Kaushik:
- - reviewing RBT process.
- - looked at retirements and pledges, and the repurposing of SLAC, prepared special RBT; 1.5M hardware only. One time infusion for the next five years.
- - Focused on retirements as we've lived under flat budgets - introducing new risks.
- - Eric prepared an RBT which was presented - there were requests for clarifications - now at round 4.
- - Monday's MB meeting the RBT was approved with "contingency". No matter - expect 1/2 of the RBT. The remaining depends on availability of funds, and getting approval from NSF. (we are restricted by NSF by a maximum change of 250k without permission - so required)
- - Things are looking good.
- - Next five years still not approved. NSF requested a revised proposal (submitted last month). There will be a review in two weeks at NSF.
- - Can start to prepare paperwork for the RBT. But cannot spend on it yet.
- - Eric is preparing letters to each of the Tier2 PI's
- - Numbers were determined based on needed hardware replacements
- MOU by Eric and Paolo, etc.
- Explicit requirements for each Tier2 for the special funds.
- Purchase as quickly as possible.
- Expect email within a few days. It will be based on the 1.5M RBT level.
- Question: can we do better with Dell, knowing we have the large purchase?
- Kaushik - Dell may setup a new portal for US ATLAS, which even BNL could take advantage. Left open the possibility of a follow-up.
- Shawn: Gary Kriegel from the original portal is still there, could be brought in.
- Compare real costs as much as possible.
- Kaushik will call Dell.
- Phone meeting on purchasing plans among Tier2's?
- Eric - prepared a short SOW for each site which summarizes how the RBT funding would be spent and delivery date.
More on procurements. Next week we need to:
1) focus on original FY16 procurements
2) Draft modified SOW's with deployment plans for the RBT funding.
3) Updates to table 4 in http://bit.ly/usatlas-capacity table
Table 4: FY16 Budget activity and capacity increments Center FY16 equipment budget ($) FY16 installed equipment purchases ($) Unspent FY16 ($) CPU capacity increase with FY16 purchases (HS06) Job slots (single logical threads) increase with FY16 purchases Storage capacity increase with FY16 purchases (TB) COMMITTED (but not installed) FY16 equipment funds ($) CPU capacity increase with COMMITTED FY16 purchases (HS06) Job slots (single logical threads) increase with COMMITTED FY16 purchases Expected date CPU available to ATLAS Storage capacity increase with COMMITTED FY16 purchases (TB) Expected date storage available to ATLAS Tier1 $2,217,000 $360,313 $1,856,687 2,450 $650,000 7526 6/30/2016 AGLT2 $250,000 $250,000 $0 MWT2 $457,491 $268,503 $188,988 15550 1344 July 25, 2016 NET2 $68,286 $68,286 576 $0 576 6/30/2016 SWT2 $260,000 $260,000 $0 WT2 $200,000 $200,000 $0 USATLAS FACILITY $3,452,777 $360,313 $2,903,476 0 0 3,026 $838,988 15550 1344 8102 USATLAS TIER2 $1,235,777 $0 $1,046,789 0 0 576 $188,988 15550 1344 576 -
13:25
→
13:35
Capacity News: Procurements & Retirements 10m
-
13:35
→
13:45
Production 10mSpeaker: Mark Sosebee (University of Texas at Arlington (US))
-
13:45
→
13:50
Data Management 5mSpeaker: Armen Vartapetian (University of Texas at Arlington (US))
-
13:50
→
13:55
Data transfers 5mSpeaker: Hironori Ito (Brookhaven National Laboratory (US))
-
13:55
→
14:00
Networks 5mSpeaker: Dr Shawn McKee (University of Michigan ATLAS Group)
-
14:00
→
14:05
FAX and Xrootd Caching 5mSpeakers: Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Andrew Hanushevsky, Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:25
→
15:25
Site Reports
-
14:25
BNL 5mSpeaker: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR))
-
14:30
AGLT2 5mSpeakers: Robert Ball (University of Michigan (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
-
14:35
MWT2 5mSpeakers: David Lesny (Univ. Illinois at Urbana-Champaign (US)), Lincoln Bryant (University of Chicago (US))
Site is now running well
Power issue at UChicago site on June 6 then June 15 caused network problems
- A small network stack of 6248 was dropping 50% of network traffic
- Only compute nodes were affected
- Caused excessive SRM etc transfer errors
- Nodes taken offline
- Upgrades of firmware and configuration changes did not fix the problem
- On June 20 Lincoln and UChicago network admin addressed the problem
- Broke up stack (only two switches)
- Connected switches to DMZ switch in "top-of-rack" configuration\
- Network 100% functional
- All nodes put back online
Updated to OSG 3.3.13
- CVMFS 2.2.3
- HTCondor 8.4.7
- Updated HTCondor CE
- WN Client
Object Store
- Setup on Ceph system
- Stress testing is planned by Doug
- Lincoln working with Alex to tune for better performance
FY16 Purchases
- UChicago
- Moving forward with storage purchase
- 1.6PB addition to dCache
- dCache head node replacement
- Illinois
- 6 x C6320 chassis (4 nodes per chassis, 24 nodes)
- Xeon E5-2680 v4 (56 Logical Cores/node)
- 11.57 HS06/LC
- 1344 Logical Cores
- Adds 15,550 HS06
- 256GB (4.5GB/LC)
- 10Gb to each node
- 2 x Juniper QFX5200 switches
- 4x10Gb uplink to Campus Consolidator Switch (CSS)
- Will be replaced with 1 or 2 40GB in future
- 132x10Gb for nodes
- 264 ports total
- Will have 124 nodes at 10Gb this year
- CSS attaches to DMZ switch at 8x10Gb
- DMZ has 2x100Gb to Chicago (ESnet, I2, LHCONE)
- 4x10Gb uplink to Campus Consolidator Switch (CSS)
- 6 x C6320 chassis (4 nodes per chassis, 24 nodes)
- Indiana
- Fred getting final quotes from Dell (having problems getting final pricing)
- 16 Servers with Broadwell cpu
- 896 logical cores
- 10K HS06
- 256GB (4.5GB/LC)
- A small network stack of 6248 was dropping 50% of network traffic
-
14:40
NET2 5mSpeaker: Prof. Saul Youssef (Boston University (US))
We had a space token reporting problem noticed and fixed this morning.
The timeout errors from McGill have disappeared. I'm not totally sure why they have disappeared, but there were zero length files at McGill and I understand that this might have been the source of the problem rather than a networking issue.
Harvard has nearly completely migrated to condor-ce, we're 100% condor in APF (I believe). We're still working on a few associated local issues, but that is basically done.
Smooth sailing otherwise. The site has not consistently been full with PanDA production but OSG has mostly been filling up the gaps.
- 14:45
-
14:50
SWT2-UTA 5mSpeaker: Patrick Mcguigan (University of Texas at Arlington (US))
UTA:
We are in the process of testing our science DMZ with a connection to LHCOne. UTA is currently advertising a different subnet from the Tier 2 for LHCOne. When completed we will not only get LHCOne access, but we will have greater bandwidth.
UTA_SWT2:
No issues to report
SWT2_CPB:
We seem to be having a problem with pilots not picking up jobs even when there are many activated jobs in our queues. We are investigating.
-
14:55
WT2 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:25
-
15:25
→
15:30
AOB 5m
-
13:00
→
13:15