Aug 21 – 25, 2017
University of Washington, Seattle
US/Pacific timezone

Design and Execution of make-like Distributed Analyses

Aug 21, 2017, 6:10 PM
Auditorium (Alder Hall)


Alder Hall

Oral Track 1: Computing Technology for Physics Research Track 1: Computing Technology for Physics Research


Robert Fischer (Rheinisch-Westfaelische Tech. Hoch. (DE))


In particle physics, workflow management systems are primarily used as
tailored solutions in dedicated areas such as Monte Carlo production.
However, physicists performing data analyses are usually required to
steer their individual workflows manually, which is time-consuming and
often leads to undocumented relations between particular workloads. We
present a generic analysis design pattern that copes with the
sophisticated demands of end-to-end HEP analyses. The approach presents
a paradigm shift from executing parts of the analysis to defining the
analysis. The clear interface and dependencies between individual workloads then
enables a make-like execution.

Our tools allow to specify arbitrary workloads and dependencies between
them in a lightweight and scalable structure. Further features are
multi-user support, automated dependency resolution and error
handling, central scheduling, and status visualization. The WLCG
infrastructure is supported including CREAM-CE, DCAP, SRM and GSIFTP.
Due to the open structure, additional computing resources, such as local
computing clusters or Dropbox storage, can be easily added and
supported. Computing jobs execute their payload, which may be any
executable or script, in a dedicated software environment. Software
packages are installed as required, and input data is retrieved on demand.

The management system is explored by a team performing ttbb and ttH
cross section measurements.

Primary authors

Benjamin Fischer (Rheinisch-Westfaelische Tech. Hoch. (DE)) Marcel Rieger (Rheinisch-Westfaelische Tech. Hoch. (DE)) Martin Erdmann (Rheinisch-Westfaelische Tech. Hoch. (DE)) Ralf Florian Von Cube (Rheinisch-Westfaelische Tech. Hoch. (DE)) Robert Fischer (Rheinisch-Westfaelische Tech. Hoch. (DE))

Presentation materials

Peer reviewing