Distributed parallel processing analysis framework for Belle II and Hyper Suprime-Cam
Presented by Mr. Sogo MINEO on 25 Feb 2010 from 14:50 to 15:15
Type: Parallel Talk
Track: Computing Technology for Physics Research
The real time data analysis at next generation experiments is a challenge because of their enormous data rate and size. The Belle II experiment, the upgraded Belle experiment, requires to manage a data amount of O(100) times the current Belle data size collected at more than 30kHz. A sophisticated data analysis is required for the efficient data reduction in the high level trigger farm in addition to the offline analysis. On the other hand, a telescope survey with Hyper Suprime-Cam at Subaru Observatory for the search of dark energy also needs to handle a large number of CCD images whose size is comparable with that of Belle II. The feed-back of the measurement parameters obtained by the real time data processing has never been performed in the past where the parameter tuning entirely relies on an empirical method. We are now developing a new software framework named "roobasf" to be shared both by Belle II and Hyper Suprime-Cam. The framework has the well-established software-bus architecture and the object persistency interface with ROOT IO. In order to achieve the required real-time performance, the parallel processing technique is widely used to utilize a huge number of network-connected PCs with multi-core CPUs. The parallel processing is performed not only in the trivial event-by-event manner, but also in the pipeline of the application software modules which are dynamically placed on many PCs. The object data flow over the network is implemented using the Message Passing Interface (MPI) which also provides the system-wide control scheme. The framework adopts Python as the user interface language. The detailed design and the development status of the framework is presented at the conference.