Speaker
Description
Summary
Virtually all Data Acquisition Systems (DAQ) for nuclear and particle physics experiments use a large number of Field Programmable Gate Arrays (FPGAs) for data transport and more complex tasks as pattern recognition and data reduction. All these FPGAs in a large system have to share a common state like a trigger number or an epoch counter to keep the system synchronized for a consistent event/epoch building. Additionally, the collected data has to be transported with high bandwidth, optionally via the ubiquitous Ethernet protocol. Furthermore, the FPGAs' internal states and configuration memories have to be accessed for control and monitoring purposes.
Another requirement for a modern DAQ-network is the fault-tolerance for intermittent data errors in the form of automatic retransmission of faulty data. As FPGAs suffer from Single Event Effects when exposed to ionizing particles, the system has to deal with failing FPGAs. Taking all these requirements into account, the TrbNet protocol was developed.
Three virtual channels are merged on one physical medium: With the highest priority the trigger/epoch information is transported. The data channel is second in the priority order, while the control channel is the last. Combined with a small frame size of 80 bit guarantees a low latency data transport can: A system with 100 front-ends can be built with a one-way latency of 2.2us.
The user interface consists of simple interfaces only: All communication details are handled by the encapsulated TrbNet end-point.
A TrbNet hub is part of the network concept to build tree like network structures. Additionally, it serves as a data combining and forwarding unit. It features a fault tolerant behaviour of the ports: If a front-end fails the port is disabled keeping the rest of the network alive.
The TrbNet-protocol was put into each of the 550 FPGAs of the HADES-Upgrade project and has been successfully used during the HADES Au+Au campaign in April 2012. With a 2M/s Au beam and 3% interaction ratio the accepted trigger rates are 10kHz while 150MBytes/s are written to storage (benchmarks: 700 MByte/s, 60kHz, limited by other electronics). Due to the micro-structure of the beam the HADES-DAQ copes with 20kHz accepted rate on small time scales. Errors are reliably mitigated via the implemented retransmission of packets and auto-shut-down of individual links. TrbNet was also used for full monitoring of the FEE status, e.g. temperatures, voltages, fill-levels of buffers, data rates.
The network stack is written in VHDL and was successfully deployed on various Lattice and Xilinx devices. The TrbNet is also used in other experiments, like the PET-scanner prototype in Coimbra, Portugal and many systems for detector developments for PANDA and CBM at FAIR. As a platform for such set-ups, e.g. for high-channel time measurement with 15ps resolution, a generic FPGA platform (TRB3) has been developed.
