Nov 4 – 8, 2019
Adelaide Convention Centre
Australia/Adelaide timezone

Reusing distributed computing software and patterns for midscale collaborative science

Nov 5, 2019, 12:15 PM
Riverbank R3 (Adelaide Convention Centre)

Riverbank R3

Adelaide Convention Centre

Oral Track 3 – Middleware and Distributed Computing Track 3 – Middleware and Distributed Computing


Benedikt Riedel (University of Wisconsin-Madison) Benedikt Riedel (University of Wisconsin-Madison)


Many of the challenges faced by the LHC experiments (aggregation of distributed computing resources, management of data across multiple storage facilities, integration of experiment-specific workflow management tools across multiple grid services) are similarly experienced by "midscale" high energy physics and astrophysics experiments, particularly as their data set volumes are increasing at comparable rates. Often these (international, multi-institution) collaborations have outgrown the computing resources offered by their home laboratories, or the capacities of any single member institution. Unlike the LHC experiments, however, these collaborations often lack the manpower required to build, integrate and operate the systems required to meet their scale. In the Open Science Grid, we have organized a team designed to support collaborative science organizations re-use proven software and patterns in distributed processing and data management, often but not restricted to software developed for the LHC. Examples are re-use of the Rucio and FTS3 software for reliable data transfer and management, XRootD for data access and caching, Ceph for large scale pre-processing storage, and Pegasus for workflow management across heterogeneous resources. We summarize experience with the VERITAS gamma ray observatory, the South Pole Telescope (CMB detector), and the XENON dark matter search experiment.

Consider for promotion No

Primary authors

Dr Paschalis Paschos (University of Chicago) Mats Rynge (ISI/USC) Benedikt Riedel (University of Wisconsin-Madison) Frank Wuerthwein (Univ. of California San Diego (US)) Robert William Gardner Jr (University of Chicago (US))

Presentation materials