US ATLAS Computing Integration and Operations

→ US/Eastern
Description
Notes and other material available in the US ATLAS Integration Program Twiki
    • 13:00 → 13:15
      Top of the Meeting 15m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Robert William Gardner Jr (University of Chicago (US))

      Kaushik:

      • - reviewing RBT process.
      • - looked at retirements and pledges, and the repurposing of SLAC, prepared special RBT; 1.5M hardware only.  One time infusion for the next five years.
      • - Focused on retirements as we've lived under flat budgets - introducing new risks.
      • - Eric prepared an RBT which was presented - there were requests for clarifications - now at round 4.
      • - Monday's MB meeting the RBT was approved with "contingency".  No matter - expect 1/2 of the RBT.  The remaining depends on availability of funds, and getting approval from NSF.  (we are restricted by NSF by a maximum change of 250k without permission - so required)
      • - Things are looking good.  
      • - Next five years still not approved.  NSF requested a revised proposal (submitted last month).  There will be a review in two weeks at NSF. 
      • - Can start to prepare paperwork for the RBT.  But cannot spend on it yet. 
      • - Eric is preparing letters to each of the Tier2 PI's
      • - Numbers were determined based on needed hardware replacements
      • MOU by Eric and Paolo, etc.
      • Explicit requirements for each Tier2 for the special funds.
      • Purchase as quickly as possible.
      • Expect email within a few days.  It will be based on the 1.5M RBT level. 
      • Question: can we do better with Dell, knowing we have the large purchase?
        • Kaushik - Dell may setup a new portal for US ATLAS, which even BNL could take advantage. Left open the possibility of a follow-up.
        • Shawn: Gary Kriegel from the original portal is still there, could be brought in.
        • Compare real costs as much as possible.
        • Kaushik will call Dell.
      • Phone meeting on purchasing plans among Tier2's? 
      • Eric - prepared a short SOW for each site which summarizes how the RBT funding would be spent and delivery date.

       

      More on procurements.  Next week we need to:

      1) focus on original FY16 procurements

      2) Draft modified SOW's with deployment plans for the RBT funding.

      3) Updates to table 4 in http://bit.ly/usatlas-capacity table

      Table 4: FY16 Budget activity and capacity increments        
      Center FY16 equipment budget ($) FY16 installed equipment purchases ($) Unspent FY16 ($) CPU capacity increase with FY16 purchases (HS06) Job slots (single logical threads) increase with FY16 purchases Storage capacity increase with FY16 purchases (TB) COMMITTED (but not installed) FY16 equipment funds ($) CPU capacity increase with COMMITTED FY16 purchases (HS06) Job slots (single logical threads) increase with COMMITTED FY16 purchases Expected date CPU available to ATLAS Storage capacity increase with COMMITTED FY16 purchases (TB) Expected date storage available to ATLAS
      Tier1 $2,217,000 $360,313 $1,856,687     2,450 $650,000       7526 6/30/2016
      AGLT2 $250,000   $250,000       $0          
      MWT2 $457,491   $268,503       $188,988 15550 1344 July 25, 2016    
      NET2 $68,286   $68,286     576 $0       576 6/30/2016
      SWT2 $260,000   $260,000       $0          
      WT2 $200,000   $200,000       $0          
      USATLAS FACILITY $3,452,777 $360,313 $2,903,476 0 0 3,026 $838,988 15550 1344   8102  
      USATLAS TIER2 $1,235,777 $0 $1,046,789 0 0 576 $188,988 15550 1344   576  

       

       

       

       

       

    • 13:25 → 13:35
      Capacity News: Procurements & Retirements 10m
    • 13:35 → 13:45
      Production 10m
      Speaker: Mark Sosebee (University of Texas at Arlington (US))
    • 13:45 → 13:50
      Data Management 5m
      Speaker: Armen Vartapetian (University of Texas at Arlington (US))
    • 13:50 → 13:55
      Data transfers 5m
      Speaker: Hironori Ito (Brookhaven National Laboratory (US))
    • 13:55 → 14:00
      Networks 5m
      Speaker: Dr Shawn McKee (University of Michigan ATLAS Group)
    • 14:00 → 14:05
      FAX and Xrootd Caching 5m
      Speakers: Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Andrew Hanushevsky, Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))
    • 14:25 → 15:25
      Site Reports
      • 14:25
        BNL 5m
        Speaker: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR))
      • 14:30
        AGLT2 5m
        Speakers: Robert Ball (University of Michigan (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
      • 14:35
        MWT2 5m
        Speakers: David Lesny (Univ. Illinois at Urbana-Champaign (US)), Lincoln Bryant (University of Chicago (US))

        Site  is now running well

         

        Power issue at UChicago site on June 6 then June 15 caused network problems

        • A small network stack of 6248 was dropping 50% of network traffic
          • Only compute nodes were affected
          • Caused excessive SRM etc transfer errors
          • Nodes taken offline
          • Upgrades of firmware and configuration changes did not fix the problem
        • On June 20 Lincoln and UChicago network admin addressed the problem
          • Broke up stack (only two switches)
          • Connected switches to DMZ switch in "top-of-rack" configuration\
          • Network 100% functional
          • All nodes put back online

         

        Updated to OSG 3.3.13

        • CVMFS 2.2.3
        • HTCondor 8.4.7
        • Updated HTCondor CE
        • WN Client

         

        Object Store

        • Setup on Ceph system
        • Stress testing is planned by Doug
        • Lincoln working with Alex to tune for better performance

         

        FY16 Purchases

        • UChicago
          • Moving forward with storage purchase
          • 1.6PB addition to dCache
          • dCache head node replacement
        • Illinois
          • 6 x C6320 chassis (4 nodes per chassis, 24 nodes)
            • Xeon E5-2680 v4 (56 Logical Cores/node)
            • 11.57 HS06/LC
            • 1344 Logical Cores
            • Adds 15,550 HS06
            • 256GB (4.5GB/LC)
            • 10Gb to each node
          • 2 x Juniper QFX5200 switches
            • 4x10Gb uplink to Campus Consolidator Switch (CSS)
              • Will be replaced with 1 or 2 40GB in future
            • 132x10Gb for nodes
              • 264 ports total
              • Will have 124 nodes at 10Gb this year
            • CSS attaches to DMZ switch at 8x10Gb
            • DMZ has 2x100Gb to Chicago (ESnet, I2, LHCONE)
        • Indiana
          • Fred getting final quotes from Dell (having problems getting final pricing)
          • 16 Servers with Broadwell cpu
            • 896 logical cores
            • 10K HS06
            • 256GB (4.5GB/LC)

         

      • 14:40
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))

         

        We had a space token reporting problem noticed and fixed this morning. 

        The timeout errors from McGill have disappeared.  I'm not totally sure why they have disappeared, but there were zero length files at McGill and I understand that this might have been the source of the problem rather than a networking issue.

        Harvard has nearly completely migrated to condor-ce, we're 100% condor in APF (I believe).  We're still working on a few associated local issues, but that is basically done.

        Smooth sailing otherwise.  The site has not consistently been full with PanDA production but OSG has mostly been filling up the gaps.

         

      • 14:45
        SWT2-OU 5m
        Speaker: Dr Horst Severini (University of Oklahoma (US))

        - no issues

        - Lucille had brief outage because A/C broke in machine room; back now

        - continue work on new OU_OSCER_ATLAS integration now

         

      • 14:50
        SWT2-UTA 5m
        Speaker: Patrick Mcguigan (University of Texas at Arlington (US))

        UTA:

        We are in the process of testing our science DMZ with a connection to LHCOne.  UTA is currently advertising a different subnet from the Tier 2 for LHCOne.   When completed we will not only get LHCOne access, but we will have greater bandwidth.

         

        UTA_SWT2:

        No issues to report

         

        SWT2_CPB:

        We seem to be having a problem with pilots not picking up jobs even when there are many activated jobs in our queues.  We are investigating.

      • 14:55
        WT2 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
    • 15:25 → 15:30
      AOB 5m