Speaker
Description
Summary 500 words
The data acquisition systems at the European X-ray Free Electron Laser (XFEL) facility will have to cope with data rates and beam structure timing comparable to those in particle physics experiments and employ similar components and techniques. There are currently 3 large pixel detectors under construction. Each detector has a one Mega-pixel 2D sensor array, with larger systems being envisaged.
The Train Builder DAQ system being developed at the STFC Rutherford Appleton Laboratory interfaces to any of the 2D detectors and will merge the image fragments from the detector links and from up to 512 X-ray pulses within each XFEL bunch train before sending complete “movies” from each detector to a farm of PCs for final processing and analysis.
XFEL generates trains of 3,000 light pulses separated by 220 ns with a train repetition rate of 10 Hz. The maximum number of images stored in the front end readout pipelines is ~512, giving a data size per train of 1 GByte. There is an inter train gap time of ~99 ms which is used by the front ends to transfer data to the Train Builder and the maximum data rate from each detector is therefore ~10 GBytes/sec. A common interface between the detectors and the Train Builder is provided using 10 Gbps SFP+ optical links.
The Train Builder exploits the regular time structure of the data flow from the XFEL detectors by using a time switched multiplexing architecture to build complete sequences of images, or “movies”, from each detector for each bunch train. This is achieved with an input stage comprising of FPGAs receiving data fragments from the detector links which then feed an analogue cross-point switch operating in a barrel shifter pattern at the train repetition rate. The data streams emerging from the switch are collected in an output stage of FPGAs which accumulate the completed movies and transmit them to the PC farm. In order to implement the barrel shift architecture deep data buffers are required at the input and output stages. This is achieved in external memory modules attached to the FPGAs.
The system will be implemented using the Advanced Telecom ATCA standard. The Train Builder demonstrator is an ATCA board with eight 10G SFP+ optical links housed on standard VITA57 pluggable FMC mezzanine cards. Each board has four Virtex-5 FPGAs for data processing each of which are attached to dual 1 GByte DDR2 SODIMMs providing the data buffering. Dual PowerPC 440 micro-controllers, embedded in the FPGA, are used to manage the DDR2 memory controller DMA engines. The analogue cross-point switch is an 80x80 way device operating at up to 6.5 Gbps. The fast data transmission will employ 10G UDP and TCP/IP based protocols implemented directly in the FPGA logic.
Only one variant of the hardware is required as each board can be configured by reprogramming the FPGAs to operate as an input receiving image fragments from the detectors or as an output sending completed images to the PC farm. In order to scale to larger systems additional cross-point switches will be implemented on dedicated switch boxes housed in the ATCA crate. These will be connected to multiple ATCA boards via Rear Transition Modules using high speed parallel optical links.
The first ATCA demonstrator boards are due for manufacture in Q4/2011.
