Speaker
Description
The EP-SFT Stacks team provides LCG software stack nightly builds for 50 different combinations of STACK and PLATFORM.
Historically, the publication of nightlies on CVMFS utilized a single machine, resulting in extended processing times that lasted until the end of the working day. To speed up the publication, and enable the distribution of a larger set of platforms every day, the LCG stack publication was moved to distributing the CVMFS gateway publication across multiple machines, thus reducing the CVMFS installation bottleneck.
Because we are constrained by the requirement of keeping the historic file system hierarchy, not all parts of the publication can be parallelized.
This talk will describe the development of a Jenkins pipeline to parallelize the publication of the LCG software stacks. It will explain how we use the Jenkins locking mechanism in combination with gateway leases to manage parallelization and mitigate potential file system conflicts. It will show in detail which parts of our publication stages, such as the installation of packages, remain partially serialized due to their structure in the file system, while other stages, such as the creation and installation of platform-specific views, are now executed in parallel.
