Focus on:  Hide Contributions
Login

US ATLAS Computing Integration and Operations

→ US/Eastern
virtual room (your office)

virtual room

your office
Description
Notes and other material available in the US ATLAS Integration Program Twiki
    • 13:00 → 13:15
      Top of the Meeting 15m
      Speakers: Michael Ernst, Robert William Gardner Jr (University of Chicago (US))
      Meeting Minutes
      US ATLAS Jira Issues
      US ATLAS S&C technical planning meeting notes
    • 13:15 → 13:25
      Capacity News: Procurements & Retirements 10m
      Capacities
    • 13:25 → 13:35
      Production 10m
      Speaker: Mark Sosebee (University of Texas at Arlington (US))
      shift-summary-12_16_15.pdf
      shift-summary-12_23_15.pdf
    • 13:35 → 13:40
      Data Management 5m
      Speaker: Armen Vartapetian (University of Texas at Arlington (US))
    • 13:40 → 13:45
      Data transfers 5m
      Speaker: Hironori Ito (Brookhaven National Laboratory (US))
    • 13:45 → 13:50
      Networks 5m
      Speaker: Dr Shawn McKee (University of Michigan ATLAS Group)
    • 13:50 → 13:55
      FAX and Xrootd Caching 5m
      Speakers: Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Andrew Hanushevsky (Unknown), Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))
      Caching xrootd proxy - update

      FAX by Wei

      1. DPM needs its own N2N for better performance. Andy worked on the Xrootd side, David Smith worked on extending the existing N2N (there is no conflict with the existing N2N use cases). 

      2. Needs all sites to export physical storage path in FAX - required in order to use the RUCIO metalink

    • 14:15 → 15:15
      Site Reports
      • 14:15
        BNL 5m
        Speaker: Michael Ernst

        Smooth operations, mostly reprocessing jobs over the course of the last week

        • Very demanding re memory (24 GB of RSS per 8-core job), resulting in WNs to swap, observed 100k/s swap in/out at times but jobs completed successfully
        • BNL has been top contributor in this campaign

        Disk storage

        • Still in the process of populating recently acquired disk storage, primarily the secondary SMR based storage systems 
        • Evaluating next generation RAID systems for FY16 procurement
        • Ceph object store now fully redundant re power, eliminates the last single point of failure   

        Cloud activities

        • Preparing central infrastructure for second large scale test on AWS; aiming at using ~100k cores

         

      • 14:20
        AGLT2 5m
        Speakers: Robert Ball (University of Michigan (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
      • 14:25
        MWT2 5m
        Speakers: David Lesny (Univ. Illinois at Urbana-Champaign (US)), Lincoln Bryant (University of Chicago (US))

        Site is running well

        • Full of Atlas jobs (MCORE, SCORE, Analy and Opport)
        • Some problems with high mem MCORE last week
          • Using 3-3.5GB per core RSS
          • Excessive swapping on lower memory nodes
          • Inefficient
          • Caused GPFS problems on low mem Illinois nodes
          • These jobs are now gone

         

        Adjust space token

        • Armen pointed out that LOCALGROUPDISK was nearly full
        • Used the last of the retired PRODDISK + old OSG space token (~40TB)
        • May need to move more from USERDISK and GROUPDISK

         

        MWT2 Accounting verified

        • WLCG numbers (from OSG Gratia) are very close to Panda (larger by ~2%)
        • Panda numbers come from Atlas Dashboard
        • Difference from local Tier3 AtlasConnect jobs running on Tier2 cores.

         

        Networking

        • Illinois ⇔ Indiana high latency
        • Taking inefficient path via I2
        • Illinois networking is aware and will work with IndianaGigaPOP
        • Need to advertise direct route between two sites via OmniPOP

         

        New hardware status

        • UChicago
          • 18 Ceph Servers
          • Rolling online of new servers / upgrade of existing servers.
        • Indiana
          • 24 R630 (E5-250, 128GB) have been delivered
          • Waiting on PDU to provide power
        • Illinois - Done
      • 14:30
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))
      • 14:35
        SWT2-OU 5m
        Speaker: Dr Horst Severini (University of Oklahoma (US))

        running well, nothing to report

      • 14:40
        SWT2-UTA 5m
        Speaker: Patrick Mcguigan (University of Texas at Arlington (US))
      • 14:45
        WT2 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))

        smooth operation

    • 15:15 → 15:20
      AOB 5m