Oct 10 – 14, 2016
San Francisco Marriott Marquis
America/Los_Angeles timezone

Workflow Management for Complex HEP Analyses

Oct 13, 2016, 3:30 PM
1h 15m
San Francisco Marriott Marquis

San Francisco Marriott Marquis

Poster Track 7: Middleware, Monitoring and Accounting Posters B / Break

Speaker

Robert Fischer (Rheinisch-Westfaelische Tech. Hoch. (DE))

Description

We present the novel Analysis Workflow Management (AWM) that provides users with the tools and competences of professional large scale workflow systems. The approach presents a paradigm shift from executing parts of the analysis to defining the analysis.

Within AWM an analysis consists of steps. For example, a step defines to run a certain executable for multiple files of an input data collection. Each call to the executable for one of those input files can be submitted to the desired run location, which could be the local computer or a remote batch system. An integrated software manager enables automated user installation of dependencies in the working directory at the run location. Each execution of a step item creates one report for bookkeeping purposes containing error codes and output data or file references. Required files, e.g. created by previous steps, are retrieved automatically. Since data storage and run locations are exchangeable from the step's perspective, computing resources can be used opportunistically. A visualization of the workflow as a graph of the steps in the web browser provides a high-level view on the analysis. The workflow system is developed and tested alongza`side of a ttbb cross section measurement where, for instance, the event selection is represented by one step and a Bayesian statistical inference is performed by another.

The clear interface and dependencies between steps enables a make-like execution of the whole analysis.

Tertiary Keyword (Optional) Analysi tools and techniques
Secondary Keyword (Optional) Distributed workload management
Primary Keyword (Mandatory) Data processing workflows and frameworks/pipelines

Primary author

Robert Fischer (Rheinisch-Westfaelische Tech. Hoch. (DE))

Co-authors

Marcel Rieger (Rheinisch-Westfaelische Tech. Hoch. (DE)) Martin Erdmann (Rheinisch-Westfaelische Tech. Hoch. (DE)) Ralf Florian von Cube (Rheinisch-Westfaelische Tech. Hoch.(DE))

Presentation materials