Oct 10 – 14, 2016
San Francisco Marriott Marquis
America/Los_Angeles timezone

The merit of data processing application elasticity

Oct 13, 2016, 3:30 PM
1h 15m
San Francisco Marriott Marquis

San Francisco Marriott Marquis

Poster Track 5: Software Development Posters B / Break


Dr Vardan Gyurjyan (Jefferson Lab)


IO optimizations along with the vertical and horizontal elasticity of an application are essential to achieve data processing performance linear scalability. However to deploy these three critical concepts in a unified software environment presents a challenge and as a result most of the existing data processing frameworks rely on external solutions to address them. For example in a multicore environment we run multiple copies of an application to attain "synthetic" vertical scalability. We rely on complex batch processing systems (with tons of overhead) to imitate so-called horizontal scaling. IO optimizations are not addressed most of the time, because entire effort is spent to perform data processing algorithmic optimizations. Note that IO and algorithmic optimizations by nature are very different and are difficult to address them simultaneously in a tightly coupled software environment.
In this paper we present CLAS12 reconstruction and analyses (CLARA) framework based data processing application design experiences and results.
CLARA is a real-time data stream-processing framework, that implements service-oriented architecture (SOA) in a flow based programming (FBP) paradigm. The choice of a paradigm with conjunction of a publish-subscribe message-passing middleware (MPM) allows integrating above-mentioned critical requirements in a unified software framework. CLARA presents an environment for developing agile, elastic, multilingual data processing applications, presenting solutions, capable of processing large volumes of distributed data interactively.

Primary Keyword (Mandatory) Software development process and tools
Secondary Keyword (Optional) Parallelizarion
Tertiary Keyword (Optional) Distributed data handling

Primary author

Dr Vardan Gyurjyan (Jefferson Lab)


Bryan Muffit (JLAB) Carl Timmer (TJNAF) David Abbott (Jefferson Lab) Ed Jastrzembski (JLAB) Graham Heyes (JLAB) Ricardo Oyarzun (Universidad Tecnica Federico Santa Maria, Chile) Sebastian Mancilla (Departamento de Fisica-Univ. Tecnica Federico Santa Maria (UTFSM) William Gu (JLAB)

Presentation materials