HTCondor Workshop Autumn 2023

18 Sept 2023, 09:00 → 22 Sept 2023, 14:00 Europe/Paris

Auditorium Joliot Curie (IJCLab)

Helge Meinhard (CERN), Todd Tannenbaum (University of Wisconsin Madison (US)), Chris Brew (Science and Technology Facilities Council STFC (GB)), Christoph Beyer, Michel Jouvin (Université Paris-Saclay (FR))

Dear HTCondor Community,

See last update about the whereabouts here: 

https://indico.cern.ch/event/1274213/page/29408-accommodation-local-transport

We are very pleased to announce that the 2023 European HTCondor Workshop will be held from Tuesday 19th September to Friday 22nd September, at IJCLab in Orsay, France.

To allow ease of travel to and from the venue the meeting will start at on Tuesday morning and run until lunchtime on the Friday.

Registration is now open.

There will be a small workshop fee of €160 to cover lunches, teas and coffees, and a social dinner on the Thursday evening.

The workshop will be an excellent occasion for learning from the experts (the developers!) about HTCondor, exchanging with your colleagues about experiences and plans and providing your feedback to the experts. 

The HTCondor Compute Entrypoint (CE) will be covered as well as with token authentication (currently a hot topic), along with general use and administration of HTCondor.

Participation is open to all organisations (including companies) and persons interested in HTCondor (and by no means restricted to particle physics and/or academia!) If you know potentially interested persons, don't hesitate to make them aware of this opportunity.

The workshop will cover both using and administering HTCondor; topics will be chosen to best match participants' interests.

We would very much like to know about your use of HTCondor, in you project, your experience and your plans. Hence you are warmly encouraged to propose a short presentation.

If you have any questions, please contact hepix-2023condorworkshop-support@hepix.org.

We are looking forward to a rich, productive workshop.

Chris Brew (STFC - RAL) and Christoph Beyer (DESY), Co-Chairs of organising committee.

Michel Jouvin (IJCLab), Chair of the Local Organising Committee 

Todd Tannenbaum, HTCondor Technical Lead, U Wisconsin, Madison, USA

 

Monday 18 September

board meeting

final board meeting for workshop preparation

Convener: Christoph Beyer

Tuesday 19 September

1 Welcome

Speaker: Christoph Beyer

2 Welcome and housekeeping

Speaker: Michel Jouvin (Université Paris-Saclay (FR))

Logistics.pdf

3 IJCLab in a nutshell

Speakers: Valerie CHAMBERT, Valerie Chambert (Universite de Paris-Sud 11 (FR))

IJCLab-Presentation-workshopHTCCondor.pdf

Workshop Session

4 HTCSS Vocabulary, Architecture, and User View

Speaker: Todd Tannenbaum (University of Wisconsin Madison (US))

intro_and_architecture.pdf

intro_and_architecture.pptx

10:30 Coffee

Data and Security

5 Data and Security intro

Speaker: MIRON LIVNY

6 Pelican: Data Federations and HTCSS

Speaker: Brian Paul Bockelman (University of Wisconsin Madison (US))

European-HTCondor-Workshop-2023-Pelican.pdf

7 IGWN Data Story

Speakers: Peter Couvares, Mr Peter Couvares (California Institute of Technology)

Couvares LIGO-G2301968-v1.pdf

8 Token Taxonomy

mixing them up… IDTOKENS for daemons, others for jobs, issued from OAuth vs Vault vs self signed vs ...

Speaker: Brian Paul Bockelman (University of Wisconsin Madison (US))

European-HTCondor-Workshop-2023-Token-Taxonomy.pdf

12:30 Lunch

Data and Security

9 (Sci)Tokens authentication in HTCondor-CE (incl new features)

Speaker: James Frey (University of Wisconsin Madison (US))

SciTokensOnTheCE.pdf

SciTokensOnTheCE.pptx

10 Getting Your Data with HTCondor

Speaker: Brian Paul Bockelman (University of Wisconsin Madison (US))

European-HTCondor-Workshop-2023-Moving-Data .pdf

11 Data and Security Wrap Up

Speaker: MIRON LIVNY

15:30 Coffee

Lightning Talks

12 Welcome drink

La Part des Anges is an excellent and friendly Wine Bar.

Tapas/apetizers offered, drinks at your charge
Possibility to have a diner afterward on an individual basis

https://osm.org/go/0BOQg93_M?node=6574870872

Wednesday 20 September

Jobs and Access Points

13 Jobs and Access Points Introduction

Speaker: MIRON LIVNY

14 Talk of all the Tools

Speaker: Gregory Thain

whirlwind-tour.pdf

whirlwind-tour.pptx

15 Visit of ThomX and SUPRATECH facilities

2 groups of 15

10:30 Coffee

Jobs and Access Points

16 Linux tuning for large scale APs

Speaker: Gregory Thain

Tuning APs.pdf

Tuning APs.pptx

17 What Are My Jobs Doing?

Speaker: James Frey (University of Wisconsin Madison (US))

HTCondorJobInfo-CKoch.pdf

18 Whirlwind Tour: Job Placement Options

Speaker: Gregory Thain

Submit Commands.pdf

Submit Commands.pptx

12:30 Lunch

Jobs and Access Points

19 HTCondor Services for Non-Humans

Speaker: James Frey (University of Wisconsin Madison (US))

HTCondor for Non-Humans.pdf

HTCondor for Non-Humans.pptx

20 Jobs and Access Points Wrap Up

Speaker: MIRON LIVNY

HTCondor User Presentations

21 News from the DESY Clusters

With the ever changing IT and political landscapes also the HTC clusters at DESY are evolving with these changes. We will give an overview and status, what over the past year is new and where we plan to go with our compute clusters. Efficient utilization of resources has become even more pressing following recent geopolitical turmoil and with climate change becoming even more pressing, for which we are optimizing the energy profile of our user Condor cluster. Also a Cobald/Tardis Condor meta cluster is under development with the idea to use it for backfilling of untapped idle resources.

Speaker: Thomas Hartmann (Deutsches Elektronen-Synchrotron (DE))

DESYHH.pdf

22 Deployment of HTCondor at GRIF

In this communication, we are going to present the multiple HTCondor instances that have been deployed at GRIF for several years. GRIF is a distributed Tier-2 WLCG/EGI site made of four (4) different subsites (IJCLab, IRFU, LLR, LPNHE), in different locations of the Paris region. The worst network latency between the subsites is within 2-4 msec with three (3) of them connected with a 100Gbit/sec connection. In particular, a distributed HTCondor pool, with HTCondor-CE gateways, gives unified access to the IJCLab and LLR resources. IRFU and LPHNE are running independent condor pools based on an ARC-CE and HTCondor-CE gateways which are providing access to the computing resources of those sites. Future intentions and plans about the incorporation and usage of Cloud computing resources with or without Kurbenetes infrastructure will be discuss.

Speaker: Dr Emmanouil Vamvakopoulos (Université Paris-Saclay (FR))

htcondor@GRIF_2023-3.pdf

15:40 Coffee

HTCondor User Presentations

23 CERN Update

Speaker: Ben Jones (CERN)

condoreu23lxbatch.pdf

condoreu23lxbatch.pptx

24 Office Hours

Thursday 21 September

Workshop Session

25 What’s New, What’s Coming Up

Speaker: Todd Tannenbaum (University of Wisconsin Madison (US))

Whats_New_european_workshop_2023 - Copy.pdf

Whats_New_european_workshop_2023 - Copy.pptx

26 Energy Savings: Roosters and Condors, oh my!

Speaker: Todd Tannenbaum (University of Wisconsin Madison (US))

Power_Management_european_workshop_2023.pdf

Power_Management_european_workshop_2023.pptx

27 Visit of ThomX and SUPRATECH facilities

By groups of 15

10:35 Coffee

HTCondor User Presentations

28 Reaching new scales in the CMS Global pool

The computing resource needs of LHC experiments, such as CMS, are expected to continue growing significantly over the next decade, during the Run 3 and especially the HL-LHC era. The SI team manages a set of federated HTCondor pools, currently aggregating around 400k CPU cores distributed worldwide, supporting the simultaneous execution of over 200k CMS computing tasks. In order to detect and overcome performance degradation driven by scalability barriers, the SI team regularly runs tests to explore the scalability reach of our infrastructure. In this contribution, we will report on the test results for potential scalability limitations of our infrastructure.

Speaker: Marco Mascheroni (Univ. of California San Diego (US))

20230921_Reaching new scales in the CMS Global pool.pdf

29 The Virgo Data Quality Reports: an HTCondor automated framework to vet gravitational-wave candidates

Transient gravitational-wave (GW) signals have been discovered since 2015 by the LVK global network of giant, ground-based, interferometric detectors. It currently includes four instruments: the LIGO Hanford and LIGO Livingston detectors located in the USA, the Virgo detector in Italy – hosted by the European Gravitational Observatory (EGO) –, and the KAGRA detector in Japan. A key component of the search for GWs is the broadcast of a low-latency public alert to the astronomical community, each time a significant GW candidate is identified by the pipelines which jointly analyze in real time the data from all the running detectors.

These alerts require a quick, yet accurate, vetting of the quality of the corresponding data. The main input for such a decision – to confirm the alert or to retract it if the candidate is found not to be of astronomical origin – comes from the Data Quality Report (DQR), a set of predefined checks which is triggered automatically when a new candidate has been identified, and which runs on an HTCondor farm. In this contribution, we focus on the Virgo DQR framework which has been developed jointly by the Virgo Collaboration and the EGO IT department, following standards defined at the LIGO-Virgo Collaborations level in 2018-2019. It allows vetting the data acquired by the Virgo detector. After a short description of the EGO HTCondor farm and of the various software which run on it during a joint data-taking period of the LVK network, we will describe the Virgo DQR; the way a significant GW candidate leads to the generation of a global HTCondor DAG (running about 40 different checks in parallel, for a total of roughly 120 jobs); its main inputs and outputs; its performance; and finally the live monitoring system which has been developed to parse every minute the dag.dagman.out DAG log file. Similar but independent DQR frameworks are running at various LIGO computing centers to vet the LIGO Hanford and LIGO Livingston data.

Speaker: Dr Nicolas Arnaud (IJCLab (Université Paris-Saclay and CNRS/IN2P3))

202309DD_HTCondor_IJCLab.pdf

30 Energy savings by power modulation of a HTC pool

Energy prices are reaching an all time high in europe and we are speculating at the same time to get a more elaborated billing model that calculates a real time electricity price depending on the availability of electricity on the market. As the price is predictable roughly 30h in forehand it seems to be desirable to not only know more about the energy consumption of a condor pool but also to drive the energy consumption of the pool along a timeline that is given by said prices ...

Speaker: Christoph Beyer

power_modulation_condor_pool.pdf

12:30 Lunch

HTCondor User Presentations

31 Multi-tenancy HTCondor for the European Weather Cloud

The European Weather Cloud (EWC) is the cloud-based collaboration platform for meteorological application development and operations in Europe and enables the digital transformation of the European Meteorological Infrastructure. Among the services to be provided to the meteorological community, batch processing has been requested for several type of analysis using satellite data. HTCondor has been selected as the batch system for data processing and it will be provided to the users through the EWC. In this talk you will discover what is EWC and how HTCondor was customized to work in the EWC in order to facilitate utilization/share of resources across multi-tenancies and to run jobs through HTCondor. In particular, you will see how we allow users to easily join the common pool of resources that can be used by the community, how nodes are automatically deployed, how we preserve the ability to access internal networks of different tenancies via VPN, how users can rely on multi tenancies resources, running jobs securely and with close to the data.

Speaker: Francesco Murdaca (EUMETSAT)

Batch Processing on EWC - HTCondor Europe Workshop 2023 - Francesco Murdaca, Mike Grant.pdf

32 Just-in-time matching of workflows for the DUNE experiment

The DUNE experiment is a large international particle physics
project which is currently under construction at Fermilab in
Illinois and SURF in South Dakota, with prototypes at CERN. The
experiment relies on Fermilab’s investment in HTCondor and
GlideInWMS, and on the LArSoft ecosystem of applications
software. Initially data management was done with Fermilab’s SAM
system but this is gradually being replaced by other components.
MetaCat and Rucio are now in use as DUNE’s file metadata and
replica catalogues, and DUNE has developed a just-in-time
workflow management system, justIN, to replace the SAM workflow
functionality and provide higher level management of processing
requests which are carried out in GlideInWMS/HTCondor jobs. The
new system’s philosophy of matching tasks to resources as they
become available will be described. justIN provides a workflow
submission interface and then submits suitable jobs to the DUNE
HTCondor pool. Jobs call back to justIN when they eventually
start at sites, and a decision is made at that point about what
workflows to carry out on that machine and which files to
process. These decisions are based on the available memory,
processors, maximum local job duration, and the availability of
nearby files which are still to be processed as part of the
current workflows. This just-in-time approach is able to take
unplanned downtimes at sites and storages into account
immediately, as well as higher level changes such as
fluctuations in the demand from other user communities. This
system was validated during the DUNE Data Challenge 4 in late
2022 and has been used in the simulation campaigns of 2023.
justIN uses token information obtained from CILogon with users
authenticating with the Fermilab Identity Provider service. This
in turn allows users to authenticate to the justIN web dashboard
or to use the justIN command line tool to launch and manage
workflows. To enforce DUNE policies on the use of Rucio-managed
storage, justIN jobs carry out data write operations on behalf
of user supplied scripts and code, which are isolated from
higher level credentials by justIN’s use of
Singularity/Apptainer containers. Further work to increase the
integration of justiN and the new dedicated DUNE HTCondor pool
will be described.

Speaker: Andrew McNab (University of Manchester)

20230921-mcnab-dune-justin.pdf

33 The journey to modern storage.

About a year a go we had a project that needs 20PB of low performance storage 20GBs throughout. We ended up buying 40PB with 2.7TBs throughout.
I would lime to share with you the journey from open source storage to commercial storage.

Choosing the technology, creating network to support this throughout, benchamrking, understand the needs and at the end real world vs the benchmark.

I hope 20 minutes will be enough

Speaker: David Handelman

Equality,Diversity, Inclusion & Accessibility

34 Thoughts on EDIA in dHTC Cyber-Infrastructure

Making space for new groups within our existing dHTC infrastructure community is essential for "democratizing computation." This talk calls on participants to reflect on why so many research computing conferences have such homogeneous attendees. Some are long-term structural issues beyond our scope, but this should not absolve us from taking proactive measures to rectify this problem where possible.

Speaker: Matthew West

edia_htcondor_paris.odp

edia_htcondor_paris.pdf

35 Discussion on EDIA

15:30 Coffee

Show your Toolbox

36 Social Diner

Location

Friday 22 September

Pools and Execution Points

37 Pools and Execution Points Intro

Speaker: MIRON LIVNY

38 Collector Overviews

Speaker: James Frey (University of Wisconsin Madison (US))

CollectorTour.pdf

CollectorTour.pptx

39 Microarchitecture

Speaker: Gregory Thain

MicroArch.pdf

MicroArch.pptx

40 What are glideins using my cluster for? Monitoring via the CE

Speaker: Todd Tannenbaum (University of Wisconsin Madison (US))

ce_monitoring_european_workshop_2023.pdf

ce_monitoring_european_workshop_2023.pptx

10:30 Coffee

Pools and Execution Points

41 Advanced debugging with eBPF and Linux perf tools

Speaker: Gregory Thain

eBPF.pptx

42 Pools and Execution Points Wrap Up

Speaker: MIRON LIVNY

43 Wrap Up and Goodbye

Speaker: Chris Brew (Science and Technology Facilities Council STFC (GB))

HTCondor_Workshop_WrapUp.pdf

HTCondor_Workshop_WrapUp.pptx

12:30 Lunch