US ATLAS Computing Integration and Operations
-
-
13:00
→
13:15
Top of the Meeting 15mSpeakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Robert William Gardner Jr (University of Chicago (US))
-
13:25
→
13:35
Capacity News: Procurements & Retirements 10m
-
13:35
→
13:45
Production 10mSpeaker: Mark Sosebee (University of Texas at Arlington (US))
-
13:45
→
13:50
Data Management 5mSpeaker: Armen Vartapetian (University of Texas at Arlington (US))
-
13:50
→
13:55
Data transfers 5mSpeaker: Hironori Ito (Brookhaven National Laboratory (US))
-
13:55
→
14:00
Networks 5mSpeaker: Dr Shawn McKee (University of Michigan ATLAS Group)
-
14:00
→
14:05
FAX and Xrootd Caching 5mSpeakers: Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Andrew Hanushevsky, Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))
From: Alessandro Di Girolamo <Alessandro.Di.Girolamo@cern.ch>
Subject: Re: Huh?
Date: October 4, 2016 at 2:09:51 PM CDT
To: Andrej <andrej.filipcic@ijs.si>
Cc: Rob Gardner <rwg@uchicago.edu>, "atlas-mgt-adc-five (Management of ATLAS Distributed Computing)" <atlas-mgt-adc-five@cern.ch>, Simone Campana <Simone.Campana@cern.ch>
Ciao,
There were no changes in strategy wrt what we discussed one year ago: the federated access through xrootd is still very important for ATLAS, but the tree of redirectors is not, the solution can be through the Rucio meta link.
Andy was there too, he can confirm.
From who did you get the (wrong) impression that something has changed?
Ciao
---
Alessandro Di Girolamo
CERN IT
+4175411-0532
---
On Oct 4, 2016 7:51 PM, "Andrej Filipčič" <andrej.filipcic@ijs.si> wrote:
Hi Rob,
there was no explicit discussion last week on FAX nor any decisions taken. But since there is an implementation in Rucio to provide the list of all replicas for the file, for all the protocols including the xrootd, and the rucio mover is supporting multiple replicas on the client side, part of FAX functionality is provided by other means. Some people might interpret that there was a decision taken that FAX should go away.
Cheers,
Andrej
On 4. 10. 16 17:05, Simone Campana wrote:
Hi.
I followed sessions in bits and pieces, I let others summarize the discussions. I think it was discussed the possibility to access directly the xrootd doors of the sites and have the client doing the replica selection
Simone
On 4 Oct 2016, at 15:44, Rob Gardner <rwg@uchicago.edu> wrote:
Dear Simone and Ale:
There is some grumbling regarding use of “FAX”, “death to FAX”, etc., as a result of decisions last week (how convenient). What does that even mean? Is there someplace I can view a summary of those decisions?
Cheers,
Rob
-
14:25
→
15:25
Site Reports
-
14:25
BNL 5mSpeaker: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR))
-
14:30
AGLT2 5mSpeakers: Robert Ball (University of Michigan (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
All R630 WN that can be brought up at MSU prior to the installation of the new Juniper EX9208 10/40Gb card are now available. We are working with MSU IT to schedule installation of the needed Juniper firmware (this Friday, 10am). It will certainly be a few days afterwards while configurations are set before the equipment will be brought online.
We are waiting on SAS 12Gb to 6Gb cables to bring up the first of 2 storage servers at MSU. We expect the cables at any time now.
There are reports of slow network at AGLT2, but, these may have been related to the large number of ongoing file migrations as dCache storage was balanced out at the UM site. We are investigating this and other phenomena.
Computation wise, we have been full up for the past few weeks.
There is an odd issue where maxpss (Proportional Set Size) is mostly zero at AGLT2 and MWT2
https://its.cern.ch/jira/browse/ADCINFR-29
This is under investigation. It is not understood at this time. -
14:35
MWT2 5mSpeakers: David Lesny (Univ. Illinois at Urbana-Champaign (US)), Lincoln Bryant (University of Chicago (US))
Indiana
New workers have been in production for some tall, all is good.
Chicago
Network
- JunOS updated on EX9208 from R12 to R14.
- Backup routing engine (RE) failed during the update, was RMA’d.
- Needed to put all PSUs at 220v to have enough power to bring new line card online.
- Work completed this morning, new line card is online
- Configuring two EX4550s today/tomorrow
Compute
- When EX4550s configured today/tomorrow will bring new MWT2 workers online (48).
Storage
- First of seven dCache nodes is built and benchmarked.
- Following AGLT2’s recipe (much thanks)
- Should be online today or tomorrow. (768 TB usable)
Analytics cluster
- Running smoothly, http://atlas-kibana.mwt2.org:5601/app/marvel
Illinois
- 3 of 6 c6320 chassis installed (12 out of 24 nodes)
- Nodes are benchmarked and online with jobs
- HS06 of 11.4 per core with 56 cores per node
- 3 chassis were damage during racking
- Still waiting on replacement parts from Dell ("soon!")
- 3 of 6 c6320 chassis installed (12 out of 24 nodes)
-
14:40
NET2 5mSpeaker: Prof. Saul Youssef (Boston University (US))
All of our arrived new worker nodes are installed now and have been running for more than a week. There are still 5 more nodes to arrive from DELL.
Our additional 1.5 PB of useable storage has arrived, is mounted, powered and is being tested. We'll be bringing this space on-line soon.
As Mark mentioned the long standing ticket related to timeout errors in FTS transfers from McGill to NET2 since there essentially haven't been these errors for more than a month
http://egg.bu.edu/NET2%7binf:NET2%7d/gadget:Studies/section:report/2016-06/McGill_timeouts_to_NET2/
On Friday, BU networking attempted to switch our WAN routes to a new ASR at the NoX site at MGHPCC. There was a problem in this operation, causing our incoming throughput to drop drastically. It took about 1/2 of the weekend to back out of the change and recoverhttp://egg.bu.edu/NET2%7binf:NET2%7d/gadget:Studies/section:report/2016-10/WAN-problem/
Notice the usefulness of perfSonar data for debugging :)
HU availability has remained fixed. In the Sept. WLCG report, availability for HU and NET2 as a whole is 100%.
We haven't had a chance to completely convert NET2/BU to HTCONDOR-CE, but there are no problems, and we expect to finish the transition soon.
Smooth operations otherwise with both sites mainly full of ATLAS production.
- 14:45
-
14:50
SWT2-UTA 5mSpeaker: Patrick Mcguigan (University of Texas at Arlington (US))
UTA_SWT2:
Datadisk (only token on site) was over-subscribed to the point of filling all storage. ADC was notified and data was transferred out / deleted. Normal operations have been restored.
SWT2_CPB:
Provisioning of hardware progressing
Problem detected with SAM numbers for the cluster. We are investigating.
-
14:55
WT2 5mSpeaker: Wei Yang (SLAC National Accelerator Laboratory (US))
-
14:25
-
15:25
→
15:30
AOB 5m
-
13:00
→
13:15