At INFN-T1 several competing groups submit their payloads to the HTCondor pool with a high level of heterogeneity. In particular, the same group can submit both multi core and single core jobs, and the ratio between these two can change quite rapidly; this and other unpredictable user side behaviours can make difficult for HTCondor administrators to provide user groups with a satisfactory fair...
The CNAF Tier-1, composed of almost 1000 worker nodes and nearly 40000 cores, completed its migration to HTCondor more than one year ago. After having adapted existing monitoring tools (built with Sensu, Influx and Grafana) to work with the new batch system, an effort has started to collect a more rich and “condor oriented” set of metrics that are used to provide better insights on the pool...
HTCondor is an effective tool to rank and match execute resources against a set of jobs with explicit resource requirements. In the cloud, a subtly different challenge is presented: how to rank execute resource configurations that will be automatically created to run idle jobs (auto-scaled on-demand).
We describe recent work by the HTCondor team and Google Cloud to provide built-in support...
Public policy design generally targets ideal households and individuals representing average figures of the population. However, statistics only make sense when referring to large numbers, less so when we are trying to represent real people belonging to the actual population. In fact, referring to the characteristics of the average citizen, the policy maker loses the capacity to represent the...
During the last year the HTCondor pools at CERN have passed the milestone of 300K cores. In this presentation we will cover some of the operational challenges we have found and the various monitoring and automation solutions deployed to tackle them. We will review as well how we envision the evolution of the service in the coming years.
The RAL Tier-1 runs an almost 50,000 core HTCondor batch farm which supports not only the four major LHC experiments but an increasing number of other experiments in the High Energy Physics, Astronomy and Space communities. Over the last few years there has been an increasing diversification both in the types of jobs the experiments expect to run and also in the hardware available to run...
The upgrade to HTCondor 9.0 isn't as smooth as previous upgrades. This talk discusses why and what to do.
HTCondor now has an optional integration with open source Hashicorp Vault for managing Java Web Tokens (JWTs) such as Scitokens. In the integration, the condor_submit command calls out to htgettoken (developed at Fermilab) to communicate with a Vault service. Vault takes care of the Open ID Connect protocol (which is based on Oauth 2.0) to communicate with a token issuer and securely storing...
This session is intended to serve as an opportunity for administrators to show the audience how they do their work with and on HTCondor - what are the most useful tools for them to perform their work? Why are they so useful? What do they look (and feel) like?
In case of interest, the session could be split into breakouts at some point in time.
This session will not be recorded. We would...
The CMS experiment at CERN requires vast amounts of computational power in order to process, simulate and analyze the high energy particle collisions data that enables the CMS collaboration to fulfill its research program in Fundamental Physics. A worldwide-distributed infrastructure, the LHC Computing Grid (WLCG), provides the majority of these resources, along with a growing participation...
The CMS Submission Infrastructure team manages a set of HTCondor pools to provide the vast amount of computing resources that are required by CMS to perform tasks like data processing, simulation and analysis. A set of tools that enables automation of regular tasks and maintenance of the key components of the infrastructure has been introduced and refined over the years, allowing the...
Links to material to consult beforehand will be published here.
Prior to this Q&A session, if you are unfamiliar with the HTCondor Python Bindings, please work through the online Python Bindings tutorials. Use the Binder link at the following URL to launch an interactive Jupyter notebook in your web browser with the tutorials already loaded:
https://htcondor.readthedocs.io/en/v9_0/apis/python-bindings/tutorials/index.html
Links to material to consult beforehand will be published here.
Links to material to consult beforehand will be published here.
The attached slide deck was designed to (hopefully) be self-explanatory to an HTCSS administrator. Bring forth any questions to the discussion!
In silico detection of (CRISPR) spacers matching Betacoronaviridae genomes in gut metagenomics sequencing data
Leoni G.1,2, Petrillo M.2, Puertas-Gallardo A.2, Sanges R.1, Patak A.2
1. Scuola Internazionale Superiore di Studi Avanzati (SISSA), Trieste (Italy);
2. Joint Research Center (JRC), Ispra (Italy).
The CRISPR-Cas system is the major component of the prokaryotic...
