US ATLAS Computing Facility

→ US/Eastern
    • 13:00 → 13:10
      WBS 2.3 Facility Management News 10m
      Speakers: Robert William Gardner Jr (University of Chicago (US)), Dr Shawn McKee (University of Michigan ATLAS Group)
    • 13:10 → 13:20
      OSG-LHC 10m
      Speakers: Brian Lin (University of Wisconsin), Matyas Selmeci

      OSG 3.5.19 / 3.4.53

      • XRootD 4.12.3
      • HTCondor-CE 4.4.0 (3.5 only): identify misconfigured job routes during startup
      • HTCondor 8.9.7 (upcoming)

      EL 8 support

      • Aiming for full OSG Software Stack support by the end of the year
      • osg-wn-client and osg-oasis packages are all available in osg-testing.
      • Currently working on verifying XRootD, the osg-wn-client tarball, and Frontier Squid
      • Are there US-ATLAS plans for updating to EL 8?
        • OU: new EL8 (?) cluster in the fall and will need WN, XRootD storage, and CE packages
        • BNL hopes to transition starting next year
    • 13:20 → 13:35
      Topical Reports
      Convener: Robert William Gardner Jr (University of Chicago (US))
      • 13:20
        Tommy Shearer (IRIS-HEP Fellow / UM Undergraduate) 15m

        Tommy will present his IIRIS-HEP project to create use-case specific dashboards and interfaces to networking data to support users, admins, network engineers and network researchers.

    • 13:35 → 13:40
      WBS 2.3.1 Tier1 Center 5m
      Speakers: Eric Christian Lancon (CEA/IRFU,Centre d'etude de Saclay Gif-sur-Yvette (FR)), Xin Zhao (Brookhaven National Laboratory (US))
      • Enabled user namespace on WNs for singularity jobs
      • Upgrade of FTS to version 3.9.4, for support of xrootd TPC
      • Analysis jobs failure after migration to GU PQ, due to permission issues on SCRATCHDISK, fixed.
    • 13:40 → 14:00
      WBS 2.3.2 Tier2 Centers

      Updates on US Tier-2 centers

      Convener: Fred Luehring (Indiana University (US))
      • 13:40
        AGLT2 5m
        Speakers: Philippe Laurens (Michigan State University (US)), Dr Shawn McKee (University of Michigan ATLAS Group), Prof. Wenjing Wu (Computer Center, IHEP, CAS)

        Painful dcache update:
        TLDR:  updating to 6.1.5 or 5.2.22 to pick up a TPC fix caused lots of trouble.
        Current state: We have now updated our headnodes to CentOS 7 with a patched version of 5.2.22 on all pool nodes.
        We will upgrade to the next official 5.x or 6.x release.
        We still need to determine why some of our pools nodes are transferring so slowly.
        We currently suspect a switch interface dropping packets.
        For lots of details see below.

        We had a switch issue (memory corruption in one switch unit) which caused most of the system to be disconnected from LAN on 20th June because all our vmware machine links are from this switch,  The incident lasted 2 hours.

        We see occasional disconnections from our 2 gatekeepers to condor work nodes which cause condor killing these jobs,  and the disconnection from gatekeepers were caused by the high usage of swap.. We increased the RAM on them, it seems to resolve the issue
         
        We keep improving BOINC with more intelligence and easy control, during downtime, BOINC jobs fill all the job slots. 
         
         

         

         

        Details on painful dcache update:
        We started at 5.2.16 and wanted to enable fixes for TPC which were in 6.1.5 or 5.2.22 released Wednesday 3-Jun-2020.
        We upgraded to 6.1.5 on Thursday 4-jun-2020 (basically yum update dcache) and that seemed to go well.  
        However pools started to complain of possible corruption of the BerkeleyDB meta data and require a restart to work.
        We submitted a support request but didn't hear back before the weekend.
        We tried to downgrade to 5.2.22 (which still had the TPC fix we needed).  
        But then the SRM Domain would not start, complaining about missing IGTF_LOA_CEDAR
        dCache support said this was caused by running a 6.1.5 gPlazma with a 5.2.x SRM but we were not.  

        We then tried all sorts of steps:
        downgrade to original version 5.2.16
        dcache uninstall, reboot, reinstall
        upgrade back to 5.2.22
        clean and vacuum DB
        reboot all dcache systems
        Only turn on the headnode with the main dcache services and no storage node.
        All of this didn't work with SRM not starting (same error about IGTF_LOA_CEDAR).

        Since our headnodes were the only dCache servers still running SL6,
        we decided to rebuild them (in-place) as CentOS7.   
        This worked and we restored the ZFS pool hosting the DBs on both headnodes
        and reinstalled Postgresql-11 and tried to bring dCache up.
        The same problem was still present!

        At this point it had to be something in the DB.   
        Dmitry at FNAL told us how to turn on debug logging for the SRM domain
        and we found out that the last thing it did before dying was read the 'srmuser' table.
        The problem was that 6.1.5 related credential stuff was stored in the 'encoded' field
        in the srmuser table.   We manually deleted all entries refering to IGTF_LOA_CEDAR
        Then restarting SRM worked on 5.2.22.    

        However, we then started seeing the other problem again with pool sporacically stopping:
        "Pool disabled: Meta data lookup failed and a pool restart is required: (JE 7.3.7) Environment must be closed, caused by: com.sleepycat.je.ThreadInterruptedException: Environment invalid because of previous exception: (JE 7.3.7) /dcache5/pool/meta java.lang.InterruptedException THREAD_INTERRUPTED: InterruptedException may cause incorrect internal state, unable to continue. Environment is invalid and must be closed."

        With help from Tigran we installed and used btrace (Java trace utility).   
        We put Tigran's ssh key on one of the pool nodes that had the most problems.   
        Tigran setup "traps" to get the problem.  Turns out the pools issue the above message
        when they identify that a pool thread was interrupted because they can't know
        if there is some potential problem with the pool state at the point.  
        If the interrupted thread is handled correctly it doesn't cause an issue.  
        Turns out the interruptions that triggered the problem were transfers timing out.   

        Tigran provided us with a patched 5.2.22 that we put on all pools that were having problems
        It has been working find since we deployed it 11-Jun-2020.
        IN2P3 seemed to be having the same or a similar issue.
        Since then we and IN2P3 tested 2 more tentative releases from Tigran on our most sensitive storage node, one unsuccefull, one successful.

         

         

      • 13:45
        MWT2 5m
        Speakers: David Jordan (University of Chicago (US)), Judith Lorraine Stephen (University of Chicago (US))

        Finished upgrading all dCache nodes to SL/CentOS7. Adding 2PB to DATADISK and 100TB to LOCALGROUPDISK

        Fixed IPv6 routing issues from UC. Also fixed broken perfSONAR nodes at UIUC

        Ongoing rucio mover debugging with Ilija, Mario, and Tomas

        GGUS 147492: This doesn't appear to be a site issue, I have closed. Who is the correct contact for these types of tickets?

      • 13:50
        NET2 5m
        Speaker: Prof. Saul Youssef (Boston University (US))

         

        Downtime successful.  Back into production.  New 6PB online.  

      • 13:55
        SWT2 5m
        Speakers: Dr Horst Severini (University of Oklahoma (US)), Mark Sosebee (University of Texas at Arlington (US)), Patrick Mcguigan (University of Texas at Arlington (US))

        UTA

        • Submitted paperwork for new purchases:
          • ~ 3PB storage
          • 12K HSPEC
          • Several new support machines
        • Problem this weekend with moving atlas-swt2.org registrar by OIT.

        OU:

        - Mostly smooth operations, except for

        - Transfer failures because of (apparently) too many xrdadler32 processes on the proxy server (se1.oscer.ou.edu), which cause timeouts. Investigating with xrootd and fts experts.

         

    • 14:00 → 14:05
      WBS 2.3.3 HPC Operations 5m
      Speaker: Doug Benjamin (Duke University (US))

      NERSC is running fine because we switched over to premium quality of service. It costs 2 times per node hour

      otherwise we would have no throughput. Will run this way till we get back to  Uniform Charge rate line.

      work continues at TACC. will now scale up the number of nodes and number of events per job.

    • 14:05 → 14:20
      WBS 2.3.4 Analysis Facilities
      Convener: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:05
        Analysis Facilities - BNL 5m
        Speaker: William Strecker-Kellogg (Brookhaven National Lab)

        Retirement of 1400 slots from shared pool HW. Will not reduce by that much but may be some

      • 14:10
        Analysis Facilities - SLAC 5m
        Speaker: Wei Yang (SLAC National Accelerator Laboratory (US))
      • 14:15
        ATLAS ML Platform & User Support 5m
        Speaker: Ilija Vukotic (University of Chicago (US))

        One small issue with ingress controller that affected Codas instance of the ML platform. Fixed.

        Several users were "ab"using system and creating >5 jupyterlab instance.

        Everything else running fine. 

    • 14:20 → 14:40
      WBS 2.3.5 Continuous Operations
      Conveners: Ofer Rind, Robert William Gardner Jr (University of Chicago (US))
      • Still resolving issues at WT2 preventing migration to unified PQ
      • Communication issues seem to persist with ADC (declaring downtimes, dealing w/missing files)
      • Any concerns due to this weekend's upcoming CERN DB intervention?
      • 14:20
        US Cloud Operations Summary: Site Issues, Tickets & ADC Ops News 5m
        Speakers: Mark Sosebee (University of Texas at Arlington (US)), Xin Zhao (Brookhaven National Laboratory (US))
      • 14:25
        Analytics Infrastructure & User Support 5m
        Speaker: Ilija Vukotic (University of Chicago (US))

        all running smoothly. No time to work on the Tier-0 logs ingest. 

      • 14:30
        Intelligent Data Delivery R&D (co-w/ WBS 2.4.x) 5m
        Speakers: Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER), Andrew Hanushevsky (Unknown), Ilija Vukotic (University of Chicago (US)), Wei Yang (SLAC National Accelerator Laboratory (US))

        xcaches working stably. Very good cache hit rate.

        testing Grand Unified VP queue at MWT2.

    • 14:40 → 14:45
      AOB 5m