US ATLAS Computing Integration and Operations

→ US/Eastern
Description
Notes and other material available in the US ATLAS Integration Program Twiki
    • 13:00 → 13:05
      Top of the Meeting 5m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Robert William Gardner Jr (University of Chicago (US))

      Eric: Main activity is to determine the resources required to meet the 2017 capacities.

      Paolo is drafting a request for additional resources for the Tier1.  Will need a similar document for the Tier2s, to send to NSF management.  

      Organize a meeting between Tier1 and ADC people from CERN, now open to anyone in the US facility.  Circulated the doodle.  Bring together key people from ADC to brainstorm for two days, present to ATLAS issues/concerns with workflows/setups.  Working plan and collaboration between US community and ADC.  Will setup an agenda for this -- input welcome.   S&C is coming - but goal is for a more relaxed, less dense meeting outside CERN. Dec 5,6 potential dates.

    • 13:10 → 13:15
      SRM to Gridftp Transition Plans and Status 5m
      Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
    • 13:35 → 13:45
      Production 10m
      Speaker: Mark Sosebee (University of Texas at Arlington (US))
    • 13:45 → 13:50
      Data Management 5m
      Speaker: Armen Vartapetian (University of Texas at Arlington (US))
    • 13:50 → 13:55
      Data transfers 5m
      Speaker: Hironori Ito (Brookhaven National Laboratory (US))
    • 13:55 → 14:00
      Networks 5m
      Speaker: Dr Shawn McKee (University of Michigan ATLAS Group)
    • 14:00 → 14:05
      FAX and Xrootd Caching 5m
      Speakers: Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Andrew Hanushevsky, Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))
    • 14:25 → 15:25
      Site Reports
      • 14:25
        BNL 5m
        Speaker: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR))
      • 14:30
        AGLT2 5m
        Speakers: Robert Ball (University of Michigan (US)), Dr Shawn McKee (University of Michigan ATLAS Group)

        Over half of the FY 2016 and 2016 RBT funded equipment from the August purchase bundle is now installed and operational.  The total dCache storage now available at AGLT2 is 4.7PB and our total core count is at our highest ever, with 8250 online cores.

        We have received a request concerning how to deal with the new SAS-connected MD3460 storage.  Consequently we have made a Twiki page at https://www.aglt2.org/wiki/bin/view/AGLT2/MD3460Install that details how we have configured it, and gives some useful commands to use in accessing it. 

        Network issue locally (NFS mount) caused massive influx to the HTCondor spool directories, filling them on Friday/Saturday last week (twice).  This was successfully resolved.

        https deletion at site was failing because dCache version was not sufficiently recent.  This was updated Friday morning last week, resolving the issue. 

         

      • 14:35
        MWT2 5m
        Speakers: David Lesny (Univ. Illinois at Urbana-Champaign (US)), Lincoln Bryant (University of Chicago (US))

        Site  is running well

        FY16 Purchases (~49K HS06 via 4224 logical cores, 5.25PB dCache disk)

        • UChicago
          • 6 of 7 storage R730 nodes and 3460/3060e are now racked
            • Length of chassis creates problems with PDU in the back.
            • Missing 12Gb to 6Gb SAS
            • While waiting on Dell will order a pair from Amazon to get started
            • Takes about 100 hours to build a raidset (using Bobs instructions)
          • 48 R630 Compute nodes are racked
            • Connected at 10Gb to new TOR switches
            • 9208 line card just arrived to connect to TOR switches
            • System have OS loaded and just need network and benchmarking
            • Two nodes online running jobs
          • All KOS and PE1950s are now retired
        • Indiana
          • Compute Nodes - 24 R630 nodes are racked
            • All KOIs are retired
            • Needed to replace a bad 4810 switch
            • Nodes are now being built
            • Need to benchmark and put online
        • Illinois
          • 3 of 6 c6320 chassis installed (12 out of 24 nodes)
            • Nodes are benchmarked and online with jobs
            • HS06 of 11.4 per core with 56 cores per node
          • 3 chassis were damage during racking
            • Waiting on replacement parts from Dell

         

        When all equipment is installed and online, MWT2 will provide about

        • 169K HS06
        • 16368 logical cores
        • 9PB dCache disk
      • 14:40
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))

        Smooth operations since we met last.  Both BU and HU sites are consistently full.

        All replacement & FY16 hardware is purchased.  All except 5 r630s have arrived and are installed at MGHPCC.  That's 31 worker nodes and 1.5 PB useable installed, 5 more workers coming soon.  Coming online soon.  Google spreadsheet updated with the new worker nodes.

        SAM jobs at Harvard (and therefore WLCG availability reporting for Harvard) was fixed on August 23.   I've requested a "N/A" for Harvard for August, but we should be normal for September.

        In the process of fixing the SAM problem, ADC made a change that affects all atlas sites.  It used to be that if you set your CE "flavor" to HTCONDOR-CE, SAM jobs would no longer work.  This is fixed now.

        HTCONDOR-CE is also running successfully on the BU side.  Currently 25% of pilots are condor-ce.  We expect to convert completely soon.  

        There is still a problem with between Harvard HTCONDOR-CE/SLURM and Gratia causing accounting to be lost for Harvard for August.  Horst has gone through this experience and has very helpfully offered us his notes and hints which took weeks to accumulate.  We'll be attempting to fix this and catch up with the Harvard team this afternoon.  OSG has also offered to help.

        The new version of FTS has a new optimizer following the suggestions of studies I did in 2015, consulting with Marian Babik, Shawn, Thomas Javurek and Hassen Riahi. 

        http://fts3-docs.web.cern.ch/fts3-docs/docs/optimizer/optimizer.html

        http://egg.bu.edu/LHC%7Binf:LHC%7D/gadget:FTS/section:history/fts/docs/update-2015-09-15/FTS-update.pdf

        This can give a BIG improvement in FTS throughput for ATLAS/CMS/LHCb.  

      • 14:45
        SWT2-OU 5m
        Speaker: Dr Horst Severini (University of Oklahoma (US))

        - sites running well

        - RBT funds spent on 500 TB of usable xrootd storage:

          - equipment just arrived, not yet installed

        - fixed several problems with new OU_OSCER_ATLAS site:

          - provided local scratch space and made that available to environment as $OSG_WN_TMP

          - this automagically fixed both sherpa and athenaMP crashes

          - now site fully functional as far as we can see

          - Gratia reporting fixed

          - EGI to WLCG Accounting still not working, opened ticket on that

        -  space reporting json files periodically produced and updated at OU_OCHEP_SWT2 and LUCILLE

         

      • 14:50
        SWT2-UTA 5m
        Speaker: Patrick Mcguigan (University of Texas at Arlington (US))

        The final equipment ordered from the RBT/FY16 funds has been received.  Starting to working on the deployment.

        UTA's Science DMZ has come online and we are now participating in LHCOne with a new path off campus (20Gb/s for now).

         

         

      • 14:55
        WT2 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
    • 15:25 → 15:30
      AOB 5m