Speaker
Dag Larsen
(University of Silesia (PL))
Description
Currently, the NA61/SHINE data production is performed on the CERN shared batch system, an approach inherited from its predecessor NA49. New data productions are initiated by manually submitting jobs to the batch system. An effort is now under way to migrate the data production to an automatic system, on top of a fully virtualised platform based on CernVM. There are several motivations for this. From a practical perspective, it will make it easier to both initiate new data productions, and to utilise computing resources available outside CERN, e.g. from institutes participating in NA61/SHINE, as well as commercial clouds. In addition, there is a data preservation perspective.
CernVM is a Linux distribution created by CERN specifically for the needs of virtual machines. It features a small images size (~300MB), and comes in flavours tailored to specific use-cases, e.g. developers' desktop and batch systems. It supports all popular hypervisors and clouds. Additional software components beyond the basic system can be easily installed via a package manager either through automatic boot-time contextualisation or by hand. Data production software, calibration data and other experiment-specific files are easily distributed globally via the HTTP-based CernVM file system. CernVM images are built from a software database using a special build system. This will allow legacy versions of CernVM to be adapted and built for future hypervisors and other technology. This will make it possible to run legacy versions of the data production software on the legacy CernVM versions they were originally running on, but on modern hardware, thus enabling a new approach to data preservation. An additional CernVM service, CernVM On-line, is a tool to set up virtual clusters seamlessly spanning multiple physical clouds, thus allowing for cloud-based distributed computing.
The effort to create a virtualised data production infrastructure for NA61/SHINE builds on top of these tools. The NA61/SHINE data production software has been adapted to run under CernVM through CernVM file system. Both raw and reconstructed data is stored on CERN Castor, and is accessed through Xrootd. Databases are used to keep track of the data. A web-based, graphical user interface for the data production will be available. This will allow the system to present lists of both raw data as well as existing data productions. If a new data production is needed, the privileged user may choose the data, software versions, and calibrations to be used. At the start of the processing, the system will automatically create a virtual cluster matching the requirements, and at the end of the processing, it will be terminated. This is needed to free up the resources for other users. CernVM-online will be used for the management of the virtual clusters. In case a legacy version of the software is selected, a matching version of CernVM will be used. Finished jobs will be scanned for errors, and automatically resubmitted for processing if needed. Finally, the relevant databases will be updated to reflect the freshly produced data.
Primary author
Dag Larsen
(University of Silesia (PL))
