3rd EGEE User Forum

Interactive Workflow Management in int.eu.grid

13 Feb 2008, 14:40

20m

Champagne (<a href="http://www.polydome.org">Le Polydôme</a>, Clermont-Ferrand, FRANCE)

Oral Existing or Prospective Grid Services Workflow and Parallelism

Speaker

Mr Branislav Simo (Institute of Informatics, Slovak Acedmy of Sciences)

Description

Grid computing is a useful tool for complex scientific applications, enabling their execution over a large pool of resources. Many of the deployed applications are a complex workflow composed of many smaller parts. However, most of these applications appear to their users as a monolithic black box, usually driven by a complicated and finely tuned shell script. Once the job starts executing, the user has no finer control over it than being able to abort it or to wait until it finishes. The described tool is able to visualize the inner workflow of the application. The user can completely control the job during execution, can see partial results, and can even alter it while it is still running. This allows not only to associate the produced data to the job workflow, to extend it, or to shorten it, but also to interactively debug and tune the job—something that would otherwise be possible only for a domain expert, and would be more time-consuming.

3. Impact

The tool is suitable for applications for which the user may want to adapt their execution during runtime using to partial results. Instead of repeatedly trying to run, tune, debug, and change a master script of the application, the user can modify the application workflow at runtime. If the need arises, another analysis to process any interesting partial results that were computed may be added. Or, if a simulation provides uninteresting data, the rest of the workflow subtree may be cancelled, and resources shifted to other parts of the job. Any application that currently uses a shell script calling several components (binary modules or other scripts) can be easily converted to a visually controlled workflow. The workflow can then be saved, exported to an XML file, and later reused. Such reuse is very simple even for non-experts.

URL for further information:

http://www.interactive-grid.eu/

Provide a set of generic keywords that define your contribution (e.g. Data Management, Workflows, High Energy Physics)

workflow management, semantic annotation, GUI, interactivity

4. Conclusions / Future plans

This tool for interactive workflow management of complex jobs is under development within the Int.eu.grid project. It is able to visualize a workflow of application components, to change the workflow during its execution, to display partial data results, and to store and later reuse any workflow, at any stage of execution. In the future, it will be extended with additional capabilities—containers for workflow steps—that will enable closer integration with other grid tools.

1. Short overview

The grid service in development allows users to manage interactively and comfortably complex jobs composed of multiple program executions. It is a modification of a system developed previously in the project K-Wf Grid as a management tool for application composed of web and grid services. The system uses the interactive channel of the Int.eu.grid project architecture to forward commands from a GUI to the on-site workflow manager to control the job during execution.

Presentation materials

Slides

EGEE_UF_InterWF.pdf

EGEE_UF_InterWF.ppt