US ATLAS Computing Integration and Operations

→ US/Eastern
Description
Notes and other material available in the US ATLAS Integration Program Twiki
    • 13:00 → 13:15
      Top of the Meeting 15m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Robert William Gardner Jr (University of Chicago (US))

      - Please register to the ADC US facility meeting : https://indico.cern.ch/event/576969/

    • 13:25 → 13:35
      Capacity News: Procurements & Retirements 10m
    • 13:35 → 13:45
      Production 10m
      Speaker: Mark Sosebee (University of Texas at Arlington (US))
    • 13:45 → 13:50
      Data Management 5m
      Speaker: Armen Vartapetian (University of Texas at Arlington (US))
    • 13:50 → 13:55
      Data transfers 5m
      Speaker: Hironori Ito (Brookhaven National Laboratory (US))
    • 13:55 → 14:00
      Networks 5m
      Speaker: Dr Shawn McKee (University of Michigan ATLAS Group)
    • 14:00 → 14:05
      FAX and Xrootd Caching 5m
      Speakers: Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Andrew Hanushevsky, Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))
    • 14:25 → 15:25
      Site Reports
      • 14:25
        BNL 5m
        Speaker: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR))

        - HPSS update last week. Went OK. Service back in production one day early.

      • 14:30
        AGLT2 5m
        Speakers: Robert Ball (University of Michigan (US)), Dr Shawn McKee (University of Michigan ATLAS Group)

        The EX9208 40G/10G blade at MSU is now operational, as is the N4032F.  With these 2 pieces of equipment:

        • One more dCache pool server has been brought online (msufs03).  The MSU site is currently in the process of balancing files across all pool servers, a process that should run about another 2 days.  Average input rate for the past 4 days of this balancing is 1.1GB/s
        • The last 24 R630 at MSU and from our total purchase are now being provisioned and brought online.  This should be almost totally completed by COB today.

        One further dCache pool server each at MSU and UM are being held back until the end of SC16 at which time they will also be brought online.

        That is the last of this summer's purchases of any kind.

        We've been working to understand StageIn rates on various hardware using Ilija's Analytics platform at the MWT2.  A few tentative conclusions at this point are:

        • The new R630 with 2 SSD can get StageIn rates, at times, approaching 200MB/s
        • The Dell PC6248-connected Worker Nodes are abysmal, independent of the WN hardware
          • Changing the NICs to a Juniper EX4200 doubled the StageIn rate.
        • iostat saturation of the StageIn target disk results in much slower rates
        • Some 4 disk Raid-0 StageIn target disks were amazingly poor targets (still under investigation)

        This study continues.

        From Ilija last night:

        Mario run a script that changed all the DDM endpoint prefixes in AGIS and that broke all of the FAX endpoints. 

        So to have it working now, you would need either to:
        a) revert the change in AGIS and restart (remove /rucio from the end of root DDM prefix) or
        b) manually give a list of correct prefixes (e.g. xrootd.n2n.overwriteSE=
        /pnfs/pic.es/data/atlas/atlasdatadisk/,...)
        c) use the modified version of the N2N which can handle changed prefixes ( RPM is attached to this mail.)

        Unfortunately this rpm is not working.  Ilija is trying to understand the problem.

        We updated our LSM to a (small) variation of the MWT2 version.  dccp is no longer in use.

         

      • 14:35
        MWT2 5m
        Speakers: David Lesny (Univ. Illinois at Urbana-Champaign (US)), Lincoln Bryant (University of Chicago (US))

        Site  is running well but Illinois is currently offline for a monthly PM

         

        FY16 Purchases

        • UChicago
          • 2 of 7 Storage servers are now online (5260TB in dCache)
          • 4 additional will be added over the next two days (3072TB)
          • Remaining server will be next week (768TB)
          • Will need downtime in the future to move to new dCache nameserver
          • All compute nodes online
        • Indiana
          •  All compute nodes online
        • Illinois
          • Currently down for a monthly PM
          • 10 out of 12 broken nodes are now online

         

        When all equipment is installed and online, MWT2 will provide about

        • 169K HS06
        • 16432 logical cores
        • 9100TB dCache disk

         

        We are also participating in a new Science DMZ architecture test, deploying 2 new perfSONAR nodes with 100Gb, 40Gb, and 10Gb cards. 

      • 14:40
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))

         

        We're planning to go out to Holyoke to install the last 5 r630s.

        We've been testing Disk Pools on the newest 1.5PB storage.  Testing should be done and storage in space tokens soon.

        One open ticket re: ~200 jobs failing with a strange I/O errors.  Investigating.

        Both sites are full and running smoothly otherwise.

        Catching up with Bob's/ADCs simplification campaigns.  We don't use lcg-utils in our LSM.  Working on removing BNL from our FRONTIER config as requested by John De Stefano.

        NESE project (large regional CEPH) starting up.

        LHCONE being worked on by Chuck. ETA ~1 week.

         

      • 14:45
        SWT2-OU 5m
        Speaker: Dr Horst Severini (University of Oklahoma (US))

        - OU_OSCER_ATLAS site working well

          - new LSM being tested, still some problems with that, since it's trying to use lcg-utils instead of gfal2-utils; waiting to hear back from experts

        - OU_OCHEP_SWT2 Lustre problems because of hardware failure on old DDN S2A 9900

          - being investigated, in the mean time in downtime

        - LUCILLE running well, but still occasionally gets turned off temporarily because of lost heartbeat HC jobs

          - Still not understood, but unlikely to be a site issue ...

         

      • 14:50
        SWT2-UTA 5m
        Speaker: Patrick Mcguigan (University of Texas at Arlington (US))

        UTA_SWT2

        No issues, everything is running fine.

        SWT2_CPB

        We have been fighting storage issues for the last couple of weeks where effects would vary between zero-length duplicate files, problems with deletions, and issues with contacting the SRM.  The root cause is the slowness of writes on a pair of storage servers.  We think we have identified the issue on the storage servers and will implement a change later today.

        We are also running into space issues SWT2_CPB, and are preparing to deploy 400 TB of storage, but drop 100 TB of old storage

        We have started to rack and deploy storage from the RBT funded equipment.

        Network

        We suffered a network outage for a couple of hours when an issue arose in our router.  We are following up with campus staff to understand the issue.

      • 14:55
        WT2 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
    • 15:25 → 15:30
      AOB 5m