Minutes of the Tier-3 Weekly Operations Meeting on Tuesday, April 28, 2020 at 13:30 (Chicago).

Indico: https://indico.cern.ch/event/907543/

Zoom: https://ucsd.zoom.us/j/656739403

Agenda Topics:

• Round Table of Operations:

  • Doug: 

    • Tier-3 Bi-weekly meeting last friday: 

      • HTCondor upgrade Puerto Rico (only on the CE but since no physical access, will delay until after access restrictions are over) & 

      • Mississippi (upgraded HTCondor everywhere). Doug: Ask if Eduardo can look at the mitigations.

    • Baylor SAM redirector test. Doug tested access with xrdcp and it works from Colorado. Asked for log & config files, haven’t heard back yet.

    • Colorado upgraded all WN’s to CentOS 7.8

  • Carl:

    • Same as above...

• Open tickets:

  • MN squid ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145598. No response. Ping again.

• K8s deployment status and tutorials

  • Still waiting for Slate documentation on CMS xcache app. Maybe this week.

• CMS Connect: token-based authentication 

  • Global Pool Negotiator updated to 8.9.6. Schedd RL7 upgraded to 8.9.7-pre. Jobs are running!

  • Looking into SciTokens now. Pip or rpm installs, but rpm conflicts with HTCondor itself?

• HTCondor Upgrades (also see above): 

  • HTCondor CE’s not deemed necessary to upgrade urgently by OSG, but they will tell us an upgrade schedule at some point.

  • TAMU hepcms-1.umd.edu HTCondor CE schedd at 8.6.13 

  • Maryland ce01.brazos.tamu.edu HTCondor CE schedd at 8.6.13

  • Old 8.6.3 startd’s traced to an un-upgraded glideinWMS frontend, which has been upgraded since.

From the Tier-1 Operations Meeting:

• Wilson Cluster at Fermilab may become part of the institutional cluster in a few weeks, which will have 28 nodes with older GPUs and a few with newer ones. CMS actually has access to them from CMS Connect. Later will be as accessible as FermiGrid is now, so by route of OSG in the short term. No factory entries that point to FermiGrid directly. 

• Software version scans by OSG are not going to happen. Follow up with Jeny for Tier-3 sites.

• Container security: Jeny sent the following e-mail around to the Tier-1 list:

This is the link to the document I spoke about in the previous CMS T1 meeting: https://docs.google.com/document/d/1DB-rjdUG_CJAtJsUFGKo4Y4k1L8H-T4KwWK5UjqK1F8/edit?usp=sharing 

In USCMS security, we are working on a set of best security practices on containers and orchestration frameworks for site administrators, service admins and users.

I want to collect your questions, current use cases of containers, security concerns and requests. The goal is to identify what are we missing (in FNAL, CMS) and what can we cover with the work we are already doing in USCMS security.