US ATLAS Computing Integration and Operations

→ US/Eastern
Description
Notes and other material available in the US ATLAS Integration Program Twiki
    • 13:00 → 13:15
      Top of the Meeting 15m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Robert William Gardner Jr (University of Chicago (US))

      Goal today is to attempt to standardize on configs for storage and worker nodes. 

      Compute - Summary from Shawn

      • 6.6k as configured
      • based on R630, 56 slots per server
      • mobo has 2x10g, 2x1g
      • IOPs - went with 2x480G SSDs + 1 500G NLSAS
      • Most everyone should have the original quote
      • Concerns or questions?
        • Saul: worried about local disk; Shawn: 20G / job slot, perhaps would need more.
        • What is the ATLAS requirement? 30G/job?
        • As for more disk
        • R0 over 4 disks?  (higher failure rate, but its a worker node)
        • Memory configuration (for the 192G).  Each CPU has 2x32G and 2x16G.  What about 256G option? 8x32Gs. 
      • Agreement on this config with 192G and 4x480G SSDs

       

      (see info in the chat window)

       

      Storage - Summary from Shawn

      • MD3060e 6 Gbps, does not yet support 12 Gbps 
      • MD3460 does, but it adds $3.3k to the cost of each shelf
      • Alternate is to put two H830 in the server, adds negligible cost
      • Head node will need to be R730xd (2U server)
      • At least 256G of RAM, due to the amount of storage behind it

       

       

       

    • 13:25 → 13:35
      Capacity News: Procurements & Retirements 10m
    • 13:35 → 13:45
      Production 10m
      Speaker: Mark Sosebee (University of Texas at Arlington (US))

      1) Sites are generally not full with production jobs. This is a known issue - not enough jobs/tasks coming into the system. See the MC production talk at the ADC Weekly on 7/5:

      https://indico.cern.ch/event/531083/contributions/2164542/attachments/1303757/1947493/MCProd_ADC_2016July05.pdf

      2) Otherwise US cloud functioning well - few incidents raised by shifters.

    • 13:45 → 13:50
      Data Management 5m
      Speaker: Armen Vartapetian (University of Texas at Arlington (US))
    • 13:50 → 13:55
      Data transfers 5m
      Speaker: Hironori Ito (Brookhaven National Laboratory (US))
    • 13:55 → 14:00
      Networks 5m
      Speaker: Dr Shawn McKee (University of Michigan ATLAS Group)
    • 14:00 → 14:05
      FAX and Xrootd Caching 5m
      Speakers: Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Andrew Hanushevsky, Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))
    • 14:25 → 15:25
      Site Reports
      • 14:25
        BNL 5m
        Speaker: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR))
      • 14:30
        AGLT2 5m
        Speakers: Robert Ball (University of Michigan (US)), Dr Shawn McKee (University of Michigan ATLAS Group)

        A problematic NIC (or cable or switchport, not yet localized) was causing intermittent 20 minutes outages on one of our VMWare servers, that just happened to have our primary gatekeeper running on it.  This NIC was disabled until we have the time to track down the real cause of the problem, and no further VMWare interruptions have since resulted.

        In preparation for storage upgrades at UM and MSU, one dCache pool server at each site is being de-comissioned.  One is down already, msufs05, and the second, umfs07, is now 70% drained.  Replacement storage at MSU from base Tier-2 funds is now on hand, and will be brought online as soon as it can be prepped.

         

      • 14:35
        MWT2 5m
        Speakers: David Lesny (Univ. Illinois at Urbana-Champaign (US)), Lincoln Bryant (University of Chicago (US))

        Site  is running well

         

        APF problem with iut2-gk gatekeeper

        • APF not sending enough pilots to iut2-gk for analy queue
        • Jose found some "ghost" pilots
        • After cleanout, pilots resumed
        • Most likely caused by all the networking issues at IU and UC in June
        •  

        Network problem with 100Gb link at UChicago site on evening of June 25 took MWT2 offline

        • I2 reported a monitored instability problem with the 100Gb link at UChicago
          • 50% of packets off site were being droppped
          • Internally MWT2 did not have any network problems
          • SE marked as down; all Panda Qs were offline
        • Network admins at UChicago restarted Brocade 100Gb interface on morning of June 26
          • Link became stable, no droppped packets
          • SE marked online; all Panda Qs came online
          • UChicago network admins will work with Brocade as this is the second occurance

         

        Object Store

        • Setup on Ceph system
        • Stress testing is planned by Doug
          • Lincoln has provide the keys and other information
          • Still waiting on Doug to do the testing
        • Lincoln working with Alex to tune for better performance

         

        FY16 Purchases

        • UChicago
          • On Hold
        • Indiana
          • On Hold
        • Illinois - Hardware ordered last week (note - nodes must conform to Illinois Campus Cluster configuration and pricing)
          • Dell confirmed the order
          • Nodes should be delivered and online by end of the month
            • 6 x C6320 chassis (4 nodes per chassis, 24 two socket nodes)
            • Xeon E5-2680 v4 (56 logical cores/node with HT enabled)
            • 11.57 HS06/LC
            • 1344 Logical Cores
            • Adds 15,550 HS06
            • 256GB (4.5GB/LC)
            • 10Gb to each node
          • Network switches ordered (Juniper QFX5200)
            • Racks and installation will be delayed until fall
            • Electricians are no available due to summer projects on campus

         

      • 14:40
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))
      • 14:45
        SWT2-OU 5m
        Speaker: Dr Horst Severini (University of Oklahoma (US))

        - no issues

        - old OSCER cluster Boomer is about to be shut down, so we're in the process of migrating all OSCER Panda queues to the new Schooner cluster now

         

      • 14:50
        SWT2-UTA 5m
        Speaker: Patrick Mcguigan (University of Texas at Arlington (US))
      • 14:55
        WT2 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
    • 15:25 → 15:30
      AOB 5m