J. Nogiec (FERMI NATIONAL ACCELERATOR LABORATORY)
The paper describes a component-based framework for data stream processing that allows for configuration, tailoring, and run-time system reconfiguration. The system’s architecture is based on a pipes and filters pattern, where data is passed through routes between components. Components process data and add, substitute, and/or remove named data items from a data stream. They can also manipulate data streams by buffering data, compressing/decompressing individual streams, and combining, splitting, or synchronizing multiple data streams. Configurable general- purpose filters for manipulating streams, visualizing data, persisting data, and reading data from various standard data sources are supplemented with many application specific filters, such as DSP, scripting, or instrumentation-specific components. A network of pipes and filters can be dynamically reconfigured at run- time, in response to a preplanned sequence of processing steps, operator intervention, or a change in one or more data streams. Four distinctive methods supporting reconfiguration are provided by the framework: modification of data routes, management of components’ activity states, triggering processing based on the content of the data, or the use of source addressing in components. The framework can be used to build static data stream processing applications such as monitoring or data acquisition systems as well as self-adjusting systems that would adapt their processing algorithm, presentation layer, or data persistency layer in response to changes in input data streams.