Speaker
Description
The TrainBuilder is an ATCA based data acquisition system developed at the STFC Rutherford Appleton Laboratory to provide readout for each of three Mega-pixel detectors at the European-XFEL Hamburg. Each Train Builder system constructs over 5,000 detector images per second using FPGAs with DDR2 data buffering and an analogue crosspoint switch architecture; thereby processing 10 GBytes/sec of raw image data. The TrainBuilder I/O links operate with 10 Gigabit Ethernet protocols implemented in FPGA logic. The first TrainBuilder was delivered to Eu-XFEL in August 2016 and three are now being used to commission detectors for first X-Ray beams later this year.
Summary
The TrainBuilder is an Advanced Telecom ATCA based custom data acquisition system developed at the STFC Rutherford Appleton Laboratory. It provides a common readout system for Mega-pixel detectors at the European-XFEL (X-Ray Free Electron Laser) facility in Hamburg. Each detector outputs up to 10 GBytes/sec of raw data distributed over multiple 10 Gbps SFP+ optical links. The TrainBuilder system merges detector link image fragments from up to 500 X-ray pulses in each XFEL bunch train every 100 msec using an analogue crosspoint switch and sends the complete detector movies of images to a farm of PCs. The TrainBuilder data links operate with 10GbE IP based protocols implemented in FPGA logic.
The TrainBuilder exploits the regular time structure of the data flow from the XFEL detectors by using a time switched multiplexing architecture to build complete sequences of images from each detector for each bunch train. This is achieved with an input stage comprising of FPGAs receiving data fragments from the detector links which then feed an analogue cross-point switch operating in a barrel shifter pattern at the train repetition rate of 10Hz. The data streams emerging from the switch are collected in an output stage of FPGAs which accumulate the completed movies and transmit them to a farm of PCs. In order to implement the barrel shift architecture deep data buffers are required at the input and output stages. This is achieved in DDR2 memory modules attached to the FPGAs.
Each ATCA crate instruments one Megapixel detector and contains four TrainBuilder I/O boards together with a central TrainBuilder Switch board. Each I/O board has eight 10G SFP+ optical links housed on VITA57 standard pluggable FMC mezzanine cards. The board has four Virtex-5 FPGAs for data processing each of which are attached to dual 2 GByte DDR2 SODIMMs providing the deep data buffering. Dual PowerPC 440 micro-controllers, embedded in the FPGA, are used to manage the DDR2 memory controller DMA engines. An additional static ram QDRII on each FPGA provides off chip memory for image processing.
The Switch board contains a 160x160 way analogue cross-point switch device which can operate at up to 6.5 Gbps. The I/O boards are connected to the Switch via Rear Transition modules using Infiniband standard cables using the Xilinx Aurora serial protocol.
The fast data transmission from the detectors and outputs to the PC farm employs 10GbE UDP/IP based protocols implemented in the FPGA logic. Half the links are configured as inputs receiving image fragments from sub-modules of the detectors and half as outputs sending complete trains of assembled images to the PC farm.
The first complete TrainBuilder crate system was delivered to the European-XFEL in Hamburg in August 2016 and three are now being used to commission detectors before the first X-Ray beams for experiments expected later this year.
