Minutes of the U.S. CMS Weekly Tier-3 operations meeting on Tuesday, April 21, 2020 

Indico: https://indico.cern.ch/event/907542/

Zoom Coordinates: https://ucsd.zoom.us/j/656739403

Attending: James, Dave, Carl, Doug, Kenyi

Agenda Topics

Round table of operations

• Doug: 

  • No progress with Baylor SAM redirector tests. They have opened access for Doug. Config typo somewhere?

  • Colorado - easing up stay-at-home next week in some cities.

• Carl: 

  • Ticket 145598 - Minnesota squid upgrade. NTR. Carl will poke them.

• Kenyi: 

  • Tier-3 k8s deployment: Kenyi is starting to write documentation. 

  • SLATE team is still writing the CMS xcache app, maybe ready next week.

  • CMS Connect: 

    • Will be a guinea pig to token-based authentication, so need HTCondor 8.9.6. Got hit by the matchmaking failure, so he reverted to 8.8.8. Known problem will be discussed in tomorrow's HTcondor call: https://indico.cern.ch/event/911575/

    • ACTION ITEM: James reply to e-mail from Rob Gardner (about a SLATE k8s tutorial for Tier-2 admins) that we should assume that Tier-2 site admins know nothing about k8s packages or installations to start with, so would have not already installed OKD et al. - DONE

HTCondor Upgrade Status

The queries (below) are actually checking the glidein factory tarball startd, not the startd in the batch system at the site itself. This would need an instrumented pilot job to check.

Action items:

• UMD: Doug will verify the HTCondor version is (see below) by logging in.

• Dave & James will inquire at OSG Production meeting the upgrade plans for hosted-CE’s and tarballs.

Results of HTCondor CE schedd and glidein tarball Queries

  condor_status -schedd -pool :9619 -af CondorVersion| sort | uniq -c

  condor_status -pool :9619 -af CondorVersion GLIDEIN_Factory | sort | uniq -c

Summary of results:

• About ½ of OSG pilots are running 8.6.3 tarballs and ½ 8.8.8. Upgrade cycling its way through?

• TAMU CE’s schedd is running 8.6.13

• UMD CE’s schedd is running 

• Hosted CE’s:

  • hosted-ce35 is running 8.8.7

  • hosted-ce34 is running 8.8.6

  • hosted-ce26 and hosted-ce16 are running 8.6.13

Detailed Results:

kodiak-ce.baylor.edu

schedd:

      1 $CondorVersion: 8.8.8 Mar 20 2020 PackageID: 8.8.8-1 $

startd's:

     44 $CondorVersion: 8.6.3 May 08 2017 BuildID: 404928 $ OSG

     32 $CondorVersion: 8.8.8 Feb 17 2020 BuildID: 496171 $ CERN-Prod

     30 $CondorVersion: 8.8.8 Feb 17 2020 BuildID: 496171 $ OSG

heposg01.colorado.edu

schedd:

      1 $CondorVersion: 8.8.8 Mar 20 2020 PackageID: 8.8.8-1 $

startd's:

    207 $CondorVersion: 8.6.3 May 08 2017 BuildID: 404928 $ OSG

    287 $CondorVersion: 8.8.8 Feb 17 2020 BuildID: 496171 $ CERN-Prod

   1134 $CondorVersion: 8.8.8 Feb 17 2020 BuildID: 496171 $ OSG

    245 $CondorVersion: 8.9.6 Feb 17 2020 BuildID: 496172 $ OSG

cmslpc-ce.fnal.gov

schedd:

      1 $CondorVersion: 8.8.8 Mar 19 2020 BuildID: 498525 PackageID: 8.8.8-1 $

startd's:

cmslpc-ce2.fnal.gov

schedd:

      1 $CondorVersion: 8.8.8 Mar 19 2020 BuildID: 498525 PackageID: 8.8.8-1 $

startd's:

deepthought.crc.nd.edu

schedd:

      1 $CondorVersion: 8.8.8 Feb 17 2020 BuildID: 496171 $

startd's:

     15 $CondorVersion: 8.6.3 May 08 2017 BuildID: 404928 $ OSG

      2 $CondorVersion: 8.8.8 Feb 17 2020 BuildID: 496171 $ CERN-Prod

hosted-ce35.grid.uchicago.edu

schedd:

      1 $CondorVersion: 8.8.7 Jan 06 2020 PackageID: 8.8.7-1 $

startd's:

     34 $CondorVersion: 8.9.6 Feb 17 2020 BuildID: 496172 $ OSG

cms-grid0.hep.uprm.edu

schedd:

      1 $CondorVersion: 8.9.6 Mar 19 2020 PackageID: 8.9.6-1 $

startd's:

     61 $CondorVersion: 8.6.3 May 08 2017 BuildID: 404928 $ OSG

    145 $CondorVersion: 8.8.8 Feb 17 2020 BuildID: 496171 $ OSG

bonner06.rice.edu

schedd:

      1 $CondorVersion: 8.8.8 Mar 20 2020 PackageID: 8.8.8-1 $

startd's:

ruhex-osgce.rutgers.edu

schedd:

      1 $CondorVersion: 8.8.8 Feb 17 2020 BuildID: 496171 $

startd's:

     38 $CondorVersion: 8.6.3 May 08 2017 BuildID: 404928 $ OSG

     14 $CondorVersion: 8.8.8 Feb 17 2020 BuildID: 496171 $ CERN-Prod

     12 $CondorVersion: 8.8.8 Feb 17 2020 BuildID: 496171 $ OSG

hosted-ce34.grid.uchicago.edu

schedd:

      1 $CondorVersion: 8.8.6 Dec 03 2019 PackageID: 8.8.6-1.1 $

startd's:

ce01.brazos.tamu.edu

schedd:

      1 $CondorVersion: 8.6.13 May 16 2019 $

startd's:

top.ucr.edu

schedd:

      1 $CondorVersion: 8.8.8 Mar 20 2020 PackageID: 8.8.8-1 $

startd's:

hepcms-1.umd.edu

schedd:

      1 $CondorVersion: 8.6.13 Dec 06 2018 $

startd's:

     15 $CondorVersion: 8.8.8 Feb 17 2020 BuildID: 496171 $ CERN-Prod

hosted-ce26.grid.uchicago.edu

schedd:

      1 $CondorVersion: 8.6.13 May 16 2019 $

startd's:

umiss001.hep.olemiss.edu

schedd:

      1 $CondorVersion: 8.8.8 Mar 20 2020 PackageID: 8.8.8-1 $

startd's:

     44 $CondorVersion: 8.6.3 May 08 2017 BuildID: 404928 $ OSG

      5 $CondorVersion: 8.8.8 Feb 17 2020 BuildID: 496171 $ CERN-Prod

     60 $CondorVersion: 8.8.8 Feb 17 2020 BuildID: 496171 $ OSG

hosted-ce16.grid.uchicago.edu

schedd:

      1 $CondorVersion: 8.6.13 Oct 30 2018 $

startd's: