14-18 October 2013
Amsterdam, Beurs van Berlage
Europe/Amsterdam timezone

The IceProd (IceCube Production) Framework

15 Oct 2013, 16:07
22m
Keurzaal (Amsterdam, Beurs van Berlage)

Keurzaal

Amsterdam, Beurs van Berlage

Oral presentation to parallel session Distributed Processing and Data Handling B: Experiment Data Processing, Data Handling and Computing Models Distributed Processing and Data Handling B: Experiment Data Processing, Data Handling and Computing Models

Speaker

Juan Carlos Diaz Velez (University of Wisconsin-Madison)

Description

IceProd is a data processing and management framework developed by IceCube Neutrino Observatory for processing of Monte Carlo simulations and data. IceProd runs as a separate layer on top of middleware and can take advantage of a variety of computing resources including grids and batch systems such as GLite, Condor, NorduGrid, PBS and SGE. This is accomplished by a set of dedicated daemons which process job submission in a coordinated fashion through the use of middleware plug-ins that serve to abstract the details of job submission and job management. IceProd can also manage complex workflow DAGs across distributed computing grids in order to optimize usage of resources. We describe several aspects of IceProd’s design including security, data integrity, scalability, throughput as well as the various challenges in each of these topics. We also discuss design aspects of a second generation IceProd, currently being tested in IceCube.

Primary author

Juan Carlos Diaz Velez (University of Wisconsin-Madison)

Presentation Materials