HTCondor Workshop Autumn 2022

Europe/Paris
Confindustria Cuneo - Casa Betania

Confindustria Cuneo - Casa Betania

Via Vittorio Bersezio, 9 12100 Cuneo CN Italy
Helge Meinhard (CERN), Todd Tannenbaum (University of Wisconsin Madison (US)), Chris Brew (Science and Technology Facilities Council STFC (GB)), Christoph Beyer
Description

We are very pleased to announce that the 2022 European HTCondor Workshop will be held from Tuesday 11th October to Friday 14th October. Save the dates!  

Due to the current easing of restrictions on gathering and travel we are planning to return to an in-person format meeting. So, we are delighted to announce that INFN-Torino have offered to host the meeting in the beautiful Italian city of Cuneo near Torino. To allow ease of travel to and from the venue the meeting will start at on Tuesday morning and run until lunchtime on the Friday.  

The workshop will be an excellent occasion for learning from the sources (the developers!) about HTCondor, exchanging with your colleagues about experiences and plans and providing your feedback to the experts. The HTCondor Compute Entrypoint (CE) will be covered as well. Participation is open to all organisations (including companies) and persons interested in HTCondor (and by no means restricted to particle physics and/or academia!) If you know potentially interested persons, don't hesitate to make them aware of this opportunity. 

The workshop will cover both using and administering HTCondor; topics will be chosen to best match participants' interests. We would very much like to know about your use of HTCondor, in you project, your experience and your plans. Hence you are warmly encouraged to propose a short presentation. 

If you have any questions, please contact hepix-2022condorworkshop-support@hepix.org . 

We are looking forward to a rich, productive workshop, and we hope to meet many of you at our first in-person workshop for three years! 

Chris Brew (STFC - RAL) and Christoph Beyer (DESY) 
Co-Chairs of organising committee. 

Todd Tannenbaum 
HTCondor Technical Lead, U Wisconsin, Madison, USA

 
 
Registration
HTCondor Workshop 2022 Registration
Participants
  • Ben Jones
  • Chris Brew
  • Christoph Beyer
  • Francesco Prelz
  • Gabriele Gaetano Fronze'
  • Gideon Bentum
  • Gregory Thain
  • Helge Meinhard
  • Ivan Glushkov
  • Jaime Frey
  • Jose Flix Molina
  • Marco Mascheroni
  • Matt Connell
  • Nikos Tsipinakis
  • Peter Couvares
  • Pradeep Jasal
  • Stefano Bagnasco
  • Stefano Dal Pra
  • Stefano Lusso
  • Thomas Hartmann
  • Todd Tannenbaum
  • +11
    • Closed Session for Organising Committee Meeting
      Conveners: Chris Brew (Science and Technology Facilities Council STFC (GB)), Christoph Beyer, Mr Gabriele Gaetano Fronze' (INFN Torino (IT) and LIGO-Virgo-Kagra Collaboration (US/IT/JP)), Gregory Thain, Miron Livny, Todd Tannenbaum (University of Wisconsin Madison (US))
    • Workshop Session
      • 1
        Welcome
        Speaker: Christoph Beyer
      • 2
        Introduction and House keeping
        Speaker: Mr Gabriele Gaetano Fronze' (INFN Torino (IT) and LIGO-Virgo-Kagra Collaboration (US/IT/JP))
      • 3
        Pointers to other HTCondor references
        Speaker: Gregory Thain
      • 4
        Whats new in HTCSS? Whats coming up?
        Speaker: Todd Tannenbaum (University of Wisconsin Madison (US))
      • 5
        Whirlwind Tour of all Condor Tools
        Speaker: Gregory Thain
      • 10:30
        Coffee Break
      • 6
        How OSG Increases Goodput with Elasticsearch
        Speaker: Todd Tannenbaum (University of Wisconsin Madison (US))
      • 7
        Exploitation of network-restricted resources at the Barcelona Supercomputer Center by CMS

        This contribution describes a number of actions and novel solutions introduced by the Spanish CMS community in order to facilitate the inclusion of BSC resources into the CMS computing infrastructure for their use by the collaboration. In particular, new HTCondor developments that allow condor daemons to interact via shared filesystems.

        Speaker: Jose Flix Molina (CIEMAT - Centro de Investigaciones Energéticas Medioambientales y Tec. (ES))
    • 12:30
      Lunch
    • The HTCondor CE
      Conveners: Gregory Thain, Todd Tannenbaum (University of Wisconsin Madison (US))
    • Workshop Session
      • 14
        Phasing Out GSI Authentication in the CMS Submission Infrastructure

        The CMS Computing Submission Infrastructure group manages and exploits a set of HTCondor pools to satisfy the experiment computing needs. The biggest of those pools, the so-called CMS Global Pool, currently aggregates nearly 400k CPU cores dynamically from pledged and opportunistic WLCG resources.
        Historically, authentication among the diverse components of the infrastructure used the Grid Security Infrastructure (GSI), based on identities and X509 certificates. However, more modern authentication standards based on capabilities and tokens have emerged over the years. The CMS Submission Infrastructure group is in the process of phasing out the GSI part of its authentication layers in favor of IDTokens and Scitokens. In this contribution we will report on the current status of this migration, and our plans for the final GSI phase out.

        Speaker: Nikos Tsipinakis (CERN)
    • Workshop Session
      • 15
        The HTC/HTCSS Data Story
        Speaker: Gregory Thain
      • 16
        Submitting sets of jobs
        Speaker: Todd Tannenbaum (University of Wisconsin Madison (US))
      • 17
        User data access and challenges

        To complement grid production workflows, the National Analysis Facility (NAF) at DESY aims to provide individual users with an easy way without hurdles to process their data fast.
        Since users are used to plain file paths, a number of storages are mounted as network mounts on the NAF HTCondor nodes. But with users showing a wide range of access patterns, we encountered occassionally problems with mounts and file access affecting the cluster stability.
        To better understand the interplay between users, kernel and file system clients, we are now working on extending our monitoring. With the already available detailed event monitoring on the storage instances, we are going to add a complementary status overview of the execution side with the aim of a consolidated view the of storage and compute clusters.

        Speakers: Christoph Beyer, Thomas Hartmann (Deutsches Elektronen-Synchrotron (DE))
      • 10:20
        Coffee Break
      • 18
        Gravitational-Wave Computing - Needs and Trends

        An Idiosyncratic Airing of Grievances, Condor-Style

        Speaker: Peter Couvares (LIGO Laboratory / Caltech)
      • 19
        Multi tenancy with htcondor

        At final we have a large unifed cluster.
        4 years ago there were three clusters sharing the same infrastructure (network and data center).
        A dedicated cluster per team.

        We decided to join forces and recreate a large cluster to better utilize the resources.

        Docker volumes able us to share compute resources and allow different storage access per team.

        In addition for that we are able to go further using docker networks and iptables. it allow us to share phisycal network without the risk of data leaks between teams.

        We will discuss about the challenges that we currently facing different job sizes, small pool for fast response and negotiation cycle.

        Speaker: David Handelman
    • Office Hours

      One-to-one sessions with the HTCondor developers to discuss your setup and your issues

    • 12:30
      Lunch
    • Certificates, Tokens, and Identity: Past, Present, and Future
    • The HTCSS Access Point Story
    • Workshop Session: Discussion on Future Challenges
    • Office Hours

      One-to-one sessions with the HTCondor developers to discuss your setup and your issues

    • 12:30
      Lunch
    • Workshop Session
    • Show your Toolbox

      An experimental open session for attendees to briefly demonstrator any tools, tops or tricks they use to manage their use of HTCondor.

    • Tour
    • Social