15–19 Apr 2024
Laboratoire Astroparticule et Cosmologie (APC) de l'Université Paris-Cité
Europe/Paris timezone

CernVM-FS 2.11: Lessons learnt from large-scale data processing at Jump Trading

16 Apr 2024, 09:00
25m
Amphithéatre Buffon (Laboratoire Astroparticule et Cosmologie (APC) de l'Université Paris-Cité)

Amphithéatre Buffon

Laboratoire Astroparticule et Cosmologie (APC) de l'Université Paris-Cité

15 rue Hélène Brion 75013 Paris France
Storage & Filesystems Storage and file systems

Speakers

Matt Harvey (Jump Trading) Valentin Volkl (CERN)

Description

The CernVM File System (CVMFS) provides the software distribution backbone for High Energy and Nuclear Physics experiments and many other scientific communities in the form of a globally available, shared, read-only filesystem. However recently, CVMFS has found major adoption beyond academia: in particular Jump Trading, an algorithmic and high-frequency trading firm, now uses CVMFS for software and data distribution at scales that surpass any prior usage, pushing the software to its limits. In this talk, we present the latest enhancements that enable this use-case, thanks to the support and contributions of Jump Trading and the close collaboration of their engineers with the CVMFS development team. Concretely, we report on the operational improvements that allow Jump Trading to deploy the latest CVMFS release for their highly parallelized workloads, and current plans for further performance gains.

Speaker release Yes

Authors

Presentation materials