Speaker
Description
Summary
The TrainBuilder is an Advanced Telecom ATCA based custom data acquisition system being developed at the STFC Rutherford Appleton Laboratory. It will provide a common readout system for the large 2D Mega-pixel detectors presently under construction for the European-XFEL facility in Hamburg. Each detector outputs up to 10 GBytes/sec of raw data distributed over multiple 10 Gbps SFP+ optical links. The TrainBuilder system will merge detector link image fragments from up to 512 X-ray pulses in each XFEL bunch train every 100 msec using an analogue crosspoint switch and send the complete detector movies of images to a farm of PCs. The TrainBuilder data links will operate with 10GbE IP based protocols implemented in FPGA logic.
The TrainBuilder exploits the regular time structure of the data flow from the XFEL detectors by using a time switched multiplexing architecture to build complete sequences of images from each detector for each bunch train. This is achieved with an input stage comprising of FPGAs receiving data fragments from the detector links which then feed an analogue cross-point switch operating in a barrel shifter pattern at the train repetition rate of 10Hz. The data streams emerging from the switch are collected in an output stage of FPGAs which accumulate the completed movies and transmit them to a farm of PCs. In order to implement the barrel shift architecture deep data buffers are required at the input and output stages. This is achieved in DDR2 memory modules attached to the FPGAs.
The TrainBuilder demonstrator is an ATCA board with eight 10G SFP+ optical links housed on VITA57 standard pluggable FMC mezzanine cards. Each board has four Virtex-5 FPGAs for data processing each of which are attached to dual 1 GByte DDR2 SODIMMs providing the deep data buffering. Dual PowerPC 440 micro-controllers, embedded in the FPGA, are used to manage the DDR2 memory controller DMA engines. An additional static ram QDRII on each FPGA provides off chip memory for fine image manipulation. The analogue cross-point switch is an 80x80 way device operating at up to 6.5 Gbps. The fast data transmission from the detectors and outputs to the PC farm will employ 10GbE UDP/IP based protocols implemented in the FPGA logic. Each board can be configured by reprogramming the FPGAs to operate as either an input unit receiving image fragments from the detectors or as an output sending completed images to the PC farm. In order to scale to larger systems we will interconnect the ATCA boards via Rear Transition Modules equipped with parallel optical links. Multiple boards would connect to a dedicated central switch card also implemented in the ATCA form factor.
The first ATCA demonstrator boards were manufactured in April 2012. The boards are now being tested at RAL and will subsequently be sent to Hamburg for slice tests of the complete XFEL DAQ system.
