Speaker
David Schultz
(University of Wisconsin-Madison)
Description
We describe the overall structure and new features of the second generation of IceProd, a data processing and management framework. IceProd was developed by the IceCube Neutrino Observatory for processing of Monte Carlo simulations and detector data, and has been a key component of the IceCube offline computing infrastructure since it was first deployed in 2006. It runs fully in user space as a separate layer on top of grid and batch systems. This is accomplished by a set of python daemons which process job workflow and maintain configuration and status information on every job before, during, and after processing. IceProd can also manage complex workflow DAGs across distributed computing grids in order to optimize usage of special resources such as GPUs.
As the second major version of IceProd, substantial improvements have been made to increase security, reliability, scalability, and ease of use. One new goal is to extend usage beyond centralized production to individual users and to facilitate large-scale individual analysis. Currently only a handful of IceCube users have access to more than one computing cluster; this version of IceProd should enable over 200 active IceCube data analyzers to transparently access distributed CPU and GPU resources. The scope of IceProd 2 also extends beyond IceCube-specific applications and can be used as a general grid computing tool.
Primary author
David Schultz
(University of Wisconsin-Madison)
