US ATLAS Computing Integration and Operations

→ US/Eastern
Description
Notes and other material available in the US ATLAS Integration Program Twiki
    • 13:00 → 13:15
      Top of the Meeting 15m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Robert William Gardner Jr (University of Chicago (US))
    • 13:25 → 13:35
      Capacity News: Procurements & Retirements 10m
    • 13:35 → 13:45
      Production 10m
      Speaker: Mark Sosebee (University of Texas at Arlington (US))

      1) Shift summaries for the past two weeks posted

      2) No carryover issues as of 6/8/16 - nice

      3) Sites are below capacity in terms of production - understood in light of insufficient numbers of requests

      4) New pilot release from Paul - see shift summary for 6/8/16

      5) Saul et al debugging networking issue between NET2 & CA/MCGILL

    • 13:45 → 13:50
      Data Management 5m
      Speaker: Armen Vartapetian (University of Texas at Arlington (US))
    • 13:50 → 13:55
      Data transfers 5m
      Speaker: Hironori Ito (Brookhaven National Laboratory (US))
    • 13:55 → 14:00
      Networks 5m
      Speaker: Dr Shawn McKee (University of Michigan ATLAS Group)
    • 14:00 → 14:05
      FAX and Xrootd Caching 5m
      Speakers: Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Andrew Hanushevsky, Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))
    • 14:25 → 15:25
      Site Reports
      • 14:25
        BNL 5m
        Speaker: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR))
      • 14:30
        AGLT2 5m
        Speakers: Robert Ball (University of Michigan (US)), Dr Shawn McKee (University of Michigan ATLAS Group)

        Nothing much to report.  Currently our Juniper router (shinano) may be experiencing a DoS attack.  This is under investigation.

         

      • 14:35
        MWT2 5m
        Speakers: David Lesny (Univ. Illinois at Urbana-Champaign (US)), Lincoln Bryant (University of Chicago (US))

        Site has been  running well except on June 6

         

        Power issue at UChicago site at 2:20 PM CDT, June 6

        • Took MWT2 offline for about 4 hours
        • All worker nodes rebooted
        • Dell Network switches got confused
          • Switches were passing data but dropping many packets
          • Cut off some KVMs and other head nodes
          • dCache SRM interface and doors on affected KVMs
          • Still doing some cleanup of affected nodes

         

        Updated to OSG 3.3.12

        • Some gatekeepers had issue reporting to BDII
        • OSG added timeout to osg-info-services which was too short (/etc/cron.d/osg-info-services)
          • Ticket logged to OSG
          • Information gathering against condor sites was very inefficient for Condor sites
          • Rewrite the Mats to improve (cut time to gather in half)
          • New gip in OSG testing repo installed on all gatekeepers
          • Will be in OSG 3.3.13

         

        Upgrade to HTCondor 8.4.6

        • Have not seen the schedd problem reported by AGLT2
        • Will watch for any issues and install 8.4.7 patch if need be

         

        CVMFS 2.2.2

        • 2.2.2 installed across all sites (client and server)
        • Bug fix for cache issue in 2.2.3 in OSG testing repo

         

      • 14:40
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))
      • 14:45
        SWT2-OU 5m
        Speaker: Dr Horst Severini (University of Oklahoma (US))
      • 14:50
        SWT2-UTA 5m
        Speaker: Patrick Mcguigan (University of Texas at Arlington (US))

        We have been having an issue with cooling in the machine room at SWT2_CPB.  The problem seems to have been resolved after the HVAC staff and Thermal Energy Plant personnel have rewritten their code for the control of chilled water.  So far, everything is looking good.

        No operational issues at UTA_SWT2.

        Working on deploying compute nodes and storage at SWT2_CPB, delayed to due to cooling issues, and some building construction.

      • 14:55
        WT2 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
    • 15:25 → 15:30
      AOB 5m