US ATLAS Computing Integration and Operations

→ US/Eastern
Description
Notes and other material available in the US ATLAS Integration Program Twiki
    • 13:00 → 13:15
      Top of the Meeting 15m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Robert William Gardner Jr (University of Chicago (US))
    • 13:25 → 13:35
      Capacity News: Procurements & Retirements 10m
    • 13:35 → 13:45
      Production 10m
      Speaker: Mark Sosebee (University of Texas at Arlington (US))
    • 13:45 → 13:50
      Data Management 5m
      Speaker: Armen Vartapetian (University of Texas at Arlington (US))
    • 13:50 → 13:55
      Data transfers 5m
      Speaker: Hironori Ito (Brookhaven National Laboratory (US))
    • 13:55 → 14:00
      Networks 5m
      Speaker: Dr Shawn McKee (University of Michigan ATLAS Group)
    • 14:00 → 14:05
      FAX and Xrootd Caching 5m
      Speakers: Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Andrew Hanushevsky, Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))

       

      From: Alessandro Di Girolamo <Alessandro.Di.Girolamo@cern.ch>

      Subject: Re: Huh?

      Date: October 4, 2016 at 2:09:51 PM CDT

      To: Andrej <andrej.filipcic@ijs.si>

      Cc: Rob Gardner <rwg@uchicago.edu>, "atlas-mgt-adc-five (Management of ATLAS Distributed Computing)" <atlas-mgt-adc-five@cern.ch>, Simone Campana <Simone.Campana@cern.ch>

       

      Ciao,

      There were no changes in strategy wrt what we discussed one year ago: the federated access through xrootd is still very important for ATLAS, but the tree of redirectors is not, the solution can be through the Rucio meta link.

      Andy was there too, he can confirm.

      From who did you get the (wrong) impression that something has changed?

      Ciao

      --- 

      Alessandro Di Girolamo

        CERN IT

        +4175411-0532

      ---

         

       

      On Oct 4, 2016 7:51 PM, "Andrej Filipčič" <andrej.filipcic@ijs.si> wrote:

       

      Hi Rob,

       

      there was no explicit discussion last week on FAX nor any decisions taken. But since there is an implementation in Rucio to provide the list of all replicas for the file, for all the protocols including the xrootd, and the rucio mover is supporting multiple replicas on the client side, part of FAX functionality is provided by other means. Some people might interpret that there was a decision taken that FAX should go away.

       

      Cheers,

      Andrej

       

      On 4. 10. 16 17:05, Simone Campana wrote:

      Hi.

      I followed sessions in bits and pieces, I let others summarize the discussions. I think it was discussed the possibility to access directly the xrootd doors of the sites and have the client doing the replica selection

       

      Simone

       

      On 4 Oct 2016, at 15:44, Rob Gardner <rwg@uchicago.edu> wrote:

       

      Dear Simone and Ale:

       

      There is some grumbling regarding use of “FAX”, “death to FAX”,  etc., as a result of decisions last week (how convenient). What does that even mean?  Is there someplace I can view a summary of those decisions?

       

      Cheers,

       

      Rob

       

    • 14:25 → 15:25
      Site Reports
      • 14:25
        BNL 5m
        Speaker: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR))
      • 14:30
        AGLT2 5m
        Speakers: Robert Ball (University of Michigan (US)), Dr Shawn McKee (University of Michigan ATLAS Group)

        All R630 WN that can be brought up at MSU prior to the installation of the new Juniper EX9208 10/40Gb card are now available.  We are working with MSU IT to schedule installation of the needed Juniper firmware (this Friday, 10am).  It will certainly be a few days afterwards while configurations are set before the equipment will be brought online.

        We are waiting on SAS 12Gb to 6Gb cables to bring up the first of 2 storage servers at MSU.  We expect the cables at any time now.

        There are reports of slow network at AGLT2, but, these may have been related to the large number of ongoing file migrations as dCache storage was balanced out at the UM site.  We are investigating this and other phenomena.

        Computation wise, we have been full up for the past few weeks.

        There is an odd issue where maxpss (Proportional Set Size) is mostly zero at AGLT2 and MWT2
        https://its.cern.ch/jira/browse/ADCINFR-29
        This is under investigation.  It is not understood at this time.

         

      • 14:35
        MWT2 5m
        Speakers: David Lesny (Univ. Illinois at Urbana-Champaign (US)), Lincoln Bryant (University of Chicago (US))

        Indiana

        New workers have been in production for some tall, all is good.

         

        Chicago

        Network 

            - JunOS updated on EX9208 from R12 to R14. 

            - Backup routing engine (RE) failed during the update, was RMA’d. 

            - Needed to put all PSUs at 220v to have enough power to bring new line card online.

            - Work completed this morning, new line card is online 

            - Configuring two EX4550s today/tomorrow 

        Compute

            - When EX4550s configured today/tomorrow will bring new MWT2 workers online (48).

        Storage 

            - First of seven dCache nodes is built and benchmarked. 

            - Following AGLT2’s recipe (much thanks)

            - Should be online today or tomorrow. (768 TB usable)

        Analytics cluster

            - Running smoothly, http://atlas-kibana.mwt2.org:5601/app/marvel

         

        Illinois

        • 3 of 6 c6320 chassis installed (12 out of 24 nodes)
          • Nodes are benchmarked and online with jobs
          • HS06 of 11.4 per core with 56 cores per node
        • 3 chassis were damage during racking
          • Still waiting on replacement parts from Dell ("soon!")

         

         

      • 14:40
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))

        All of our arrived new worker nodes are installed now and have been running for more than a week.   There are still 5 more nodes to arrive from DELL.

        Our additional 1.5 PB of useable storage has arrived, is mounted, powered and is being tested. We'll be bringing this space on-line soon.

        As Mark mentioned the long standing ticket related to timeout errors in FTS transfers from McGill to NET2 since there essentially haven't been these errors for more than a month

        http://egg.bu.edu/NET2%7binf:NET2%7d/gadget:Studies/section:report/2016-06/McGill_timeouts_to_NET2/

        On Friday, BU networking attempted to switch our WAN routes to a new ASR at the NoX site at MGHPCC.  There was a problem in this operation, causing our incoming throughput to drop drastically.  It took about 1/2 of the weekend to back out of the change and recover

        http://egg.bu.edu/NET2%7binf:NET2%7d/gadget:Studies/section:report/2016-10/WAN-problem/

        Notice the usefulness of perfSonar data for debugging :)

        HU availability has remained fixed.  In the Sept. WLCG report, availability for HU and NET2  as a whole is 100%.

        We haven't had a chance to completely convert NET2/BU to HTCONDOR-CE, but there are no problems, and we expect to finish the transition soon.

        Smooth operations otherwise with both sites mainly full of ATLAS production.

      • 14:45
        SWT2-OU 5m
        Speaker: Dr Horst Severini (University of Oklahoma (US))

        - all sites running well

        - have received missing network switches for new OU_OSCER_ATLAS storage

        - hope to start installing the OS on the T630s next week

         

      • 14:50
        SWT2-UTA 5m
        Speaker: Patrick Mcguigan (University of Texas at Arlington (US))

        UTA_SWT2:

          Datadisk (only token on site) was over-subscribed to the point of filling all storage.  ADC was notified and data was transferred out / deleted.  Normal operations have been restored.

         

        SWT2_CPB:

           Provisioning of hardware progressing

           Problem detected with SAM numbers for the cluster.  We are investigating.

         

      • 14:55
        WT2 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
    • 15:25 → 15:30
      AOB 5m