10-15 March 2019
Steinmatte conference center
Europe/Zurich timezone

DataForge: declarative approach to scientific data processing automation

11 Mar 2019, 17:10
Steinmatte Plenary

Steinmatte Plenary

Oral Track 1: Computing Technology for Physics Research Track 1: Computing Technology for Physics Research


Dr Alexander Nozik (INR RAS / MIPT)


Modern data processing (acquisition, storage and analysis) requires modern tools.
One of the problems shared by existing scientific software is "scripting" approach, when user writes an imperative script which describes the stages in which data should be processed. The main deficiency of such approach is the lack of possibility to automate the process. For example one usually needs script to manipulate or even textually generate other scripts in order to run complex tasks. Also scripted interaction usually could not be easily run in parallel or on distributed or cluster systems.

The DataForge metadata processing framework remedies this problem by presenting a declarative approach to data processing. In this approach the process described as a composition of tasks with automated task tree builder based on tree-like metadata communication. DataForge allows to write flexible simple tasks (in any programming style) and then create automatically managed task graphs of any complexity.

Primary author

Dr Alexander Nozik (INR RAS / MIPT)

Presentation materials

Peer reviewing