US ATLAS Computing Integration and Operations

Wednesday 4 Jan 2017, 13:00 → 16:10 US/Eastern

Notes and other material available in the US ATLAS Integration Program Twiki

13:00 → 13:15 Top of the Meeting

Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Robert William Gardner Jr (University of Chicago (US))

13:15 → 13:20 ADC news and issues

Speakers: Robert Ball (University of Michigan (US)), Wei Yang (SLAC National Accelerator Laboratory (US))

No meetings, so no new news. --bob

 

13:20 → 13:30 Production

Speaker: Mark Sosebee (University of Texas at Arlington (US))

shift-summary-12_28_16.pdf

shift-summary-1_4_17.pdf

13:30 → 13:35 Data Management

Speaker: Armen Vartapetian (University of Texas at Arlington (US))

13:35 → 13:40 Data transfers

Speaker: Hironori Ito (Brookhaven National Laboratory (US))

13:40 → 13:45 Networks

Speaker: Dr Shawn McKee (University of Michigan ATLAS Group)

13:45 → 13:50 FAX and Xrootd Caching

Speakers: Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Andrew Hanushevsky, Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))

13:50 → 14:10 Site movers

Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))

14:10 → 14:30 OS performances testing

Speaker: Doug Benjamin (Duke University (US))

14:30 → 16:05 Site Reports

14:30 BNL

Speaker: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR))

Nothing to report over Xmas break

Xin Zhao (xzhao@bnl.gov) is from now on the primary ADC contact for BNL 

14:35 AGLT2

Speakers: Robert Ball (University of Michigan (US)), Dr Shawn McKee (University of Michigan ATLAS Group)

Smooth operations over the holiday break, with only one incident of being set offline (that seemed to be shared among multiple sites).

Closeout funds are now committed for (UM)

• 13 40-core R630
• 1xN2048 1Gb switch
• 1xS4048-ON 10Gb switch
• Miscellaneous needed cables

At MSU the following were committed

• 5 40-core R630
• 4xN2048 1Gb switches
• Two PDUs and 4 UPS to support the hardware.
• Two 40Gb LR4 optical transceivers

No AGLT2 funds remain to be spent out.

Found issue just before break with our Dec 12 OSG 3.3.18 upgrade, where gfal-copy did not work.  This was the basis of one of the rsv SE probes, which had been red since the upgrade.  This was traced to the gfal2 rpm set.  These were rolled back on January 3 and the rsv SE probe returned to the green state.

 

14:40 MWT2

Speakers: David Lesny (Univ. Illinois at Urbana-Champaign (US)), Lincoln Bryant (University of Chicago (US))

New member of MWT2 - Gregory Farr (gfarr@uchicago.edu)

 

Site  is running. Full of MCORE and Analy jobs

 

Problems at Illinois at the close of 2016

• On Dec 30, GPFS went offline due to corruption
• ICC Admins worked with IBM to find problem
• File system returned on Dec 31 at about 11PM
• MWT2 nodes were back online on Jan 1.

 

dCache updates

• New hardware for nameserver
• Will require site downtime for about a day
• Not scheduled as yet

 

Site updates

• OSG 3.3.19 being deployed

 

FY17 bridge / NSF-Columbia close-out purchases 

• UChicago - 31 x R630 servers on order (1240 logical cores, 13491 HS06)
• Indiana - 15 x R630 servers on order (600 logical cores, 6528 HS06)
• Illinois - Putting together order for 6 x C6320 nodes (336 logical cores, 3820 HS06)
• Total - 2176 LC, 23839 HS06

14:45 NET2

Speaker: Prof. Saul Youssef (Boston University (US))

News: Last CA agreement purchases are working it's way through BU purchasing:

1.5 PB storage 

2 MD3640 8TB

2 MD3060e 8TB

2 R730 servers

23 R630 worker nodes, same UTA config as before

1 S4048-ON switch

That completes our capital equipment purchases from the current CA.  All are repeat quotes from our previous purchases.

LHCONE: 

Mike O'Connor and Chuck Von Lichtenberg have a test path working.  Chuck is tracking down a problem with packet loss on the non-ESnet portion of the path.  Mike expects peering with LHCONE can come up within a day of getting the problem resolved. 

Operations:

We had smooth operations over the break with only minor issues.  Both sites have been full.

14:50 SWT2-OU

Speaker: Dr Horst Severini (University of Oklahoma (US))

14:55 SWT2-UTA

Speaker: Patrick Mcguigan (University of Texas at Arlington (US))

Purchasing:

At OU, we purchased an additional 14 condo nodes for use at OSCER.

This will provide an additional 336 slots/cores and an increase of a little over 3.5K HS06.

 

At UTA we are purchasing 50 compute nodes, the same as the nodes purchased with RBD funding.  These will be split (25/25) between UTA_SWT2 and SWT2_CPB.  At the time that we deploy these at UTA_SWT2 we will be removing the oldest machines there.  UTA_SWT2 is expected to have 109 nodes, 2320 job slots and 21.1K HS06.

SWT2_CPB will have additional nodes to deploy.  It is not certain that these can be deployed without removing existing (but old) machines.  We expect SWT2_CPB to become:

421 nodes, 8256 job slots, 85.1K HS06.

We still have a small amount of money on hand that will be used for additional infrastructure needs and a head-node, or two.

 

 

15:00 WT2

Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))

16:05 → 16:10 AOB