CernVM Workshop 2024

16 Sept 2024, 09:00 → 18 Sept 2024, 19:30 Europe/Zurich

500/1-001 - Main Auditorium (CERN)

Valentin Volkl (CERN)

The CernVM Users Workshop is held from 16 to 18 September 2024 at CERN, Geneva.

 

 

The CernVM 2024 workshop follows the previous editions held at Nikhef in 2022, virtually in February 2021, at CERN in June 2019, at CERN in January 2018, at RAL (UK) in June 2016 and at CERN in March 2015.

As usual, the workshop aims to bring together users and developers to discuss the current status of the CernVM ecosystem and the future directions, with a fresh look onto the landscape of cloud technology and software delivery.

This is planned as an in-person event with the possibility of joining remotely. If you plan to attend, either in-person or remotely, please register.

We are happy to welcome invited guest speakers from the industry on selected technology topics:

• Bernd Schubert (DDN), Maintainer of libFUSE
• Miklos Szeredi, Maintainer of FUSE
• Stephane Cance, Varnish Software
• Walid Boudebouda, Varnish Software
• Matt Harvey, JumpTrading

 

Attending the event in-person is free but for logistical reasons we would like to know how many people will be there. Let us know if you wish to change your registration.

For remote participants: please register and find the Zoom details. You are also welcome to join us on Mattermost during the workshop.

For questions or comments, please contact us at cernvm-workshop@cern.ch.

 

Monday 16 September

10:00 → 12:00 CernVM-FS Hacking Session

On-site session for discussions and coding. The core team will be present and participants are welcome to join us at the venue.

Speakers: Jakob Blomer (CERN), Laura Promberger (CERN), Valentin Volkl (CERN)

12:00 → 14:00 Lunch

14:00 → 18:00 Monday afternoon: CERN team presentations

Convener: Dave Dykstra (Fermi National Accelerator Lab. (US))

14:00 Welcome and Practicalities

Speaker: Valentin Volkl (CERN)

2024-04-16-CVMFS-Workshop-Intro.pdf

14:15 Workshop Introduction

Speakers: Gerardo Ganis (CERN), Predrag Buncic (CERN)

CernVM-17th.pdf

CernVM-17th.pptx

14:30 CernVM-FS at 15: technological retrospective

Speaker: Jakob Blomer (CERN)

cvmfs-turns-15.pdf

15:00 CernVM-FS: Status and Plans

Speaker: Valentin Volkl (CERN)

2024-09-16-CVMFS-Workshop-3.pdf

15:30 Coffea Break

16:00 Zstd: A new compression algorithm for CVMFS

Speaker: Laura Promberger (CERN)

Zstd.pdf

16:20 New overlayfs features in cvmfs_server; Fuse-t on MacOS

Speaker: Yurii Bielikov (Taras Shevchenko National University of Kyiv (UA))

CVMFS_Workshop.pdf

16:40 Efficient and fast container execution using image snapshotters

A large fraction of computing workloads in high-energy and nuclear physics is executed using software containers. For physics analysis use, such container images often have sizes of several gigabytes. Executing a large number of such jobs in parallel on different compute nodes efficiently, demands the availability and use of caching mechanisms and image loading techniques to prevent network saturation and significantly reduce startup time. Using the industry-standard containerd container runtime for pulling and running containers, enables the use of various so-called snapshotter plugins that “lazily” load container images. We present a quantitative comparison of the performance of the CVMFS, SOCI, and Stargz snapshotter plugins. Furthermore, we also evaluate the user-friendliness of such approaches and discuss how such seamlessly containerised workloads contribute to the reusability and reproducibility of physics analyses.

Speaker: Max Fatouros (Paul Scherrer Institute (CH))

Max_Fatouros_CVMFS_Workshop_2024.pdf

17:00 Optimizing CERN SFT Nightlies Publication through Parallelized CVMFS Gateway Installation

The EP-SFT Stacks team provides LCG software stack nightly builds for 50 different combinations of STACK and PLATFORM.
Historically, the publication of nightlies on CVMFS utilized a single machine, resulting in extended processing times that lasted until the end of the working day. To speed up the publication, and enable the distribution of a larger set of platforms every day, the LCG stack publication was moved to distributing the CVMFS gateway publication across multiple machines, thus reducing the CVMFS installation bottleneck.
Because we are constrained by the requirement of keeping the historic file system hierarchy, not all parts of the publication can be parallelized.
This talk will describe the development of a Jenkins pipeline to parallelize the publication of the LCG software stacks. It will explain how we use the Jenkins locking mechanism in combination with gateway leases to manage parallelization and mitigate potential file system conflicts. It will show in detail which parts of our publication stages, such as the installation of packages, remain partially serialized due to their structure in the file system, while other stages, such as the creation and installation of platform-specific views, are now executed in parallel.

Speaker: Mr Tim Ehmann (CERN EP-SFT)

CVMFS_Workshop_Presentation_Tim_Ehmann.pdf

17:20 Containerizing CVMFS publishing

Speaker: Amal Santosh Thundiyil (CERN)

Containerization of CVMFS Publishing Processes - CVMFS Workshop 2024.pdf

19:30 → 22:30 Dinner - Bains des Paquis

Tuesday 17 September

09:00 → 12:00 Tuesday morning: FUSE and Varnish guest speakers

Convener: Jakob Blomer (CERN)

09:00 New Developments in LibFUSE

Speaker: Bernd Schubert

cern-fuse-presentation.pdf

09:40 FUSE Kernel Developments: Passthrough mode

Speaker: Miklos Szeredi

FUSE Kernel Developments.pdf

10:00 Coffee Break

10:30 Use of FUSE in EOS

Speaker: Elvin Alin Sindrilaru (CERN)

EOS FUSE CVMFS 2024.pdf

10:55 Jump Trading and Varnish

Speakers: Matt Harvey (Jump Trading), Stephane Cance, Mr Walid Boudebouda

CVMFS Workshop 2024 -vanish and jump combined.pdf

CVMFS Workshop 2024 -vanish and jump combined.pdf

CVMFS Workshop 2024 -vanish and jump combined.pptx

12:00 → 13:45 Lunch Break - Visits to AD & LEIR

14:00 → 18:00 Tuesday afternoon: experiments and sites

Convener: Valentin Volkl (CERN)

14:00 CernVM-FS makes software EESSI to use

/cvmfs/software.eessi.io is the production CernVM-FS repository of the European Environment for Scientific Software Installations (EESSI, pronounced as "easy", see eessi.io/docs). After several pilot iterations, EESSI has released its first production software stack in the fall of 2023. The growing software stack already includes hundreds of software installations optimised for a variety of CPU microarchitectures (x86_64 and aarch64 CPU families) that are ready to be used by anyone on any (Linux) machine anywhere in the world in a matter of minutes.

In this talk, we will present the status of EESSI (progress we made since the last CernVM workshop in 2022) and the features we are currently working on. Particularly, we will cover our efforts to automate the building of software for multiple CPU architectures, ongoing work to add support for more CPU architectures (AMD Genoa, Intel Sapphire Rapids and Fujitsu A64FX) and GPUs (NVIDIA and AMD), introduce our growing suite of portable regression tests, describe how we collaborate with software developers (including the recently created dev.eessi.io CernVM-FS repository) as well as experiences we made through training and support activities.

We will also show the progress for RISC-V CPUs (available via the CernVM-FS repository riscv.eessi.io), work on enabling and fine-tuning EESSI on (Euro)HPC sites and discuss plans for improved monitoring.

Speaker: Thomas Röblitz

CernVM-FS_makes_software_EESSI_to_use.pdf

CernVM-FS_makes_software_EESSI_to_use.pptx

14:20 LHCb installations on CVMFS

Speaker: Ben Couturier (CERN)

CVMFS-WS-202409-LHCb.pdf

14:40 Coffee Break

14:55 CVMFS Stratum 1 Performance Benchmarks

Speaker: Dave Dykstra (Fermi National Accelerator Lab. (US))

CernVMWorkshopStratum1Performance20240917.pdf

15:15 Pelican and the Open Science Data Federation

Speaker: Brian Paul Bockelman (University of Wisconsin Madison (US))

Pelican-Overview-CVMFS-Workshop-2024.pdf

15:35 CVMFS in EUCLID

Speaker: Pierre Casenove (CNES)

CVMFS_In_Euclid.pdf

CVMFS_In_Euclid.pptx

15:55 Coffee Break

16:25 ATLAS Installations on CVMFS

Speaker: John Steven De Stefano Jr (Brookhaven National Laboratory (US))

CVMFS in ATLAS: 2024

16:45 New CernVM-FS use cases at CMS

CernVM-FS remains a central service for the distribution of the CMS Offline Software (CMSSW). Traditional use cases include the distribution of CMSSW releases and container images. It also plays a crucial role in the deployment of Integration Builds (IB) and pull request testing as part of the CMSSW CI/CD workflow.

In this contribution, we present three new use cases of CernVM-FS for CMS. Firstly, the distribution of gridpacks to accelerate Monte Carlo generation. Gridpacks are pre-computed diagrams distributed in the form of tarballs that are used as lookup files. The CernVM-FS ingest command allows the serving of already-untarred gridpacks, reducing the load of unpacking them on computing sites. Secondly, the use of opportunistic HPC resources using the singcvmfs command in the cvmfsexec package. The CMS Offline infrastructure already makes use of AMD GPUs from the LUMI HPC in Finland, thanks to FUSE pre-mounting. Finally, we are currently building the CMSSW software stack for the RISC-V architecture. To include this new architecture in the CMSSW IBs, deployment to CernVM-FS using emulation on the publisher nodes has been implemented.

These new use cases at CMS highlight the importance of CernVM-FS in streamlining workflows and improving the performance of the CMS Offline infrastructure.

Speaker: Andrea Valenzuela Ramirez (CERN)

CVMFS-CMS-AValenzuela.pdf

17:05 ALICE installations on CVMFS

Speaker: Giulio Eulisse (CERN)

G01.25.07 CVMFS and ALICE.pdf

17:25 CVMFS@GSI

GSI is and has been operating a cvmfs instance for many years.
It is used primarily to make prerequisites for physics simulations and analysis of GSI and future FAIR experiments available on our compute cluster.
These include items such as precompiled executables, Spack packages, as well as Apptainer images.
The current cvmfs setup at GSI will be outlined in this talk.

Speaker: Soren Lars Gerald Fleischer (GSI - Helmholtzzentrum fur Schwerionenforschung GmbH (DE))

2024-09-17_cvmfs.at.GSI.pdf

17:40 CVMFS-CSI: CVMFS in Kubernetes Deployments at CERN-IT

Speaker: Jack Charlie Munday

cvmfs-csi.pdf

18:30 → 20:30 CERN 70 Social Event

https://community-event.web.cern.ch/

Wednesday 18 September

09:30 → 12:30 Discussion session / Hands-On: Varnish Hands-On / Discussion Session

Convener: Valentin Volkl (CERN)

09:30 Varnish Hands-On

Speaker: Mr Walid Boudebouda

Hands-on Varnish.pdf

11:00 Coffee Break

11:20 Discussion session / Hands-On

12:00 Workshop Closing

Speaker: Valentin Volkl (CERN)