Speaker
J. Nogiec
(FERMI NATIONAL ACCELERATOR LABORATORY)
Description
The paper describes a component-based framework for data stream processing that
allows for configuration, tailoring, and run-time system reconfiguration. The
system’s architecture is based on a pipes and filters pattern, where data is passed
through routes between components. Components process data and add, substitute,
and/or remove named data items from a data stream. They can also manipulate data
streams by buffering data, compressing/decompressing individual streams, and
combining, splitting, or synchronizing multiple data streams. Configurable general-
purpose filters for manipulating streams, visualizing data, persisting data, and
reading data from various standard data sources are supplemented with many
application specific filters, such as DSP, scripting, or instrumentation-specific
components. A network of pipes and filters can be dynamically reconfigured at run-
time, in response to a preplanned sequence of processing steps, operator
intervention, or a change in one or more data streams. Four distinctive methods
supporting reconfiguration are provided by the framework: modification of data
routes, management of components’ activity states, triggering processing based on
the content of the data, or the use of source addressing in components. The
framework can be used to build static data stream processing applications such as
monitoring or data acquisition systems as well as self-adjusting systems that would
adapt their processing algorithm, presentation layer, or data persistency layer in
response to changes in input data streams.
Primary authors
J. Nogiec
(FERMI NATIONAL ACCELERATOR LABORATORY)
K Trombly-Freytag
(Fermilab)
