Focus on:  Hide Contributions
Login

US ATLAS Computing Integration and Operations

→ US/Eastern
Description
Notes and other material available in the US ATLAS Integration Program Twiki
    • 1
      Top of the Meeting
      Speakers: Eric Christian Lancon (BNL), Robert William Gardner Jr (University of Chicago (US))
    • 2
      ADC news and issues
      Speakers: Robert Ball (University of Michigan (US)), Wei Yang (SLAC National Accelerator Laboratory (US))

      Please everyone work towards getting HTCondor-CE going and using the new site mover controls.

      Also check your subcluster reporting, and make sure that your OSG HS06 value is correctly reported.  With the demise of the BDII, this will be the only way to enter your site capacities into AGIS.

       

       

    • 3
      Production
      Speaker: Mark Sosebee (University of Texas at Arlington (US))
      shift-summary-5_10_17.pdf
      shift-summary-5_3_17.pdf
    • 4
      Data Management
      Speaker: Armen Vartapetian (University of Texas at Arlington (US))
      170510_DataManagement_Armen.pdf
    • 5
      Data transfers
      Speaker: Hironori Ito (Brookhaven National Laboratory (US))
    • 6
      Networks
      Speaker: Dr Shawn McKee (University of Michigan ATLAS Group)

      OSG network services having some challenges after upgrade of underlying storage (4TB->8TB disks).

      Some records corrupted.  Plan is to upgrade from Postgresql 9.4 -> 9.6 and transfer as many records as are readable.  New production will run on 9.6 while recovery work goes on for the 9.4 original.

      New perfSONAR 4.0 may be heavier resource use?  psum01.aglt2.org worked with 4GB on 3.5.x but needed 8G on 4.0.

      WLCG workshop will have network session https://indico.cern.ch/event/609911/overview.   Who will attend?

      CWP efforts ongoing.  Network text being worked on here https://docs.google.com/document/d/164SMNC3lzZGlXfqWvY7slxVbTro2Z7rZY_3DUVQT1JM/edit?ts=59032a1e#

    • 7
      FAX and Xrootd Caching
      Speakers: Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Andrew Hanushevsky, Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))

      Andy/Ilija/Hiro are working on StashCache stability issue

      Hiro has an on going request of clever user mapping plugin for BNL Box. 

      Andy/Wei working on Inverse Name2Name to cache files (in gLFN) at Proxy cache (testing).

    • 8
      Site movers
      Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))

      1. The following sites need to set " deprecate_oldmover = True" in AGIS

      • BNL_LOCAL-condor
      • OUHEP_OSG
      • OU_OSCER_ATLAS_OPP
      • Lucille_MCORE
      • OU_OSCER_ATLAS
      • OU_OCHEP_SWT2-condor
      • OU_OSCER_ATLAS_MCORE
      • Lucille_CE
      • ANALY_OU_OCHEP_SWT2-condor

      2. The following sites need to set "use_newmover = True" and "deprecate_oldmover = True" in AGIS

      • GOOGLE_COMPUTE_ENGINE 
      • ANALY_BNL_T3-condor
      • TESTGLEXEC
      • NERSC-PDSF-sge
      • TestPilot
      • TWTEST
      • ANALY_WISC_ATLAS

      3. lsm_mover.py (in pilot) has hardcoded requirement on "srm" protocol for DDM endpoint. When "srm" is removed from a DDM, lsm_mover.py won't work. Alexey A. agreed to add "root" and "gsiftp" to the list (so a DDM endpoint will need at least one of the three: "srm", "gsiftp", "root")

    • 9
      OS performances testing
      Speaker: Doug Benjamin (Duke University (US))
    • 10
      HPCs integration
      Speaker: Taylor Childers (Argonne National Laboratory (US))
      2017-05-10-USAtlasFacilities.pdf
    • Site Reports
      • 11
        BNL
        Speaker: Xin Zhao (Brookhaven National Laboratory (US))
        • AFS phased out from T1 WNs
        • singularity (test) queues running PanDA production jobs now
        • HPSS ATLAS tape storage to be upgraded to LTO-7 this week (Thursday)
      • 12
        AGLT2
        Speakers: Robert Ball (University of Michigan (US)), Dr Shawn McKee (University of Michigan ATLAS Group)

        AGLT2 is running smoothly with no issues.  All N2048 switches are now online and operational.  Reviewing some 10Gb dCache pool connections to ensure they are optimal.

        Moved Analysis queue to SCRATCHDISK without any issues.  Token sizes adjusted accordingly.

        GROUPDISK is mostly decommissioned.  Some "ghost" files exist which must be cleaned before total removal. Space reporting no longer shows this area.

         

         

      • 13
        MWT2
        Speakers: David Lesny (Univ. Illinois at Urbana-Champaign (US)), Lincoln Bryant (University of Chicago (US))

        Site is now full of jobs and operating well

         

        Minor problems/fixes during the last two weeks

        • Appear to have dark data on SCRATCHDISK
          • space usage is nearly full and required adding 20TB
          • rucio believes we have significant less usage
          • found a "ruciotest" directory
          • Could be the dark data

         

        Will soon begin the update to OSG 3.3.24

         

        .

      • 14
        NET2
        Speaker: Prof. Saul Youssef (Boston University (US))

        We are finding that fans are failing regularly on out-of-warranty worker nodes.  Have ordered replacement parts.

        Slurm hung a couple of times on the Harvard side for as yet unknown reasons.  

        Transition from USERDISK to SCRATCHDISK done with help from Armen.

        ADC wanted adjust MCORE vs SCORE on the PanDA end, wanted to know whether we were dynamic.  Worked fine.

        Lots of activity and preparations are under way for CEPH deployment within the NESE project including a rack of out-of-warranty NET2 storage. 

      • 15
        SWT2-OU
        Speaker: Dr Horst Severini (University of Oklahoma (US))

        - power outage at LU yesterday; should be resolved this afternoon

        - working on several bugs related to SLURM/Gratia/Gracc

        - otherwise no issues

         

      • 16
        SWT2-UTA
        Speaker: Patrick Mcguigan (University of Texas at Arlington (US))

        CPB:

        • Adding compute nodes
        • Started working on HTCondor-CE test isntallation

        UTA_SWT2:

        • Nothing new to report

         

        Both Sites:

        UTA's peering with ESNet via LEARN is changing physical circuits to mitigate congestion issues observed in ESNet.  The switchover will occur on Friday 5/12 at 10am CST.  Minor impacts are expected during the change over.

      • 17
        WT2
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
    • 18
      AOB