Speaker
Description
Timing and Fast Control (TFC) system for the Compressed Baryonic Matter (CBM) experiment is being developed with focus on low and deterministic data transmission latency. This helps to avoid congestion of the free-streaming Data Acquisition System (DAQ) system during occasional data bursts caused by the expected beam intensity fluctuations. Proven in latency-optimized experimental data transport applications, the GBT-FPGA core is expected to positively contribute to the TFC system performance. In this work, the core has been integrated as the primary communication interface and its effect on transmission latency and quality of time distribution has been evaluated.
Summary (500 words)
Timing and Fast Control (TFC) system of the Compressed Baryonic Matter (CBM) experiment is being developed to ensure a common time base in the Data Acquisition (DAQ) electronics with sub-ns accuracy and protect the system from excessive data bursts caused by the expected beam intensity fluctuations.
The data in the experiment is collected by the front-end electronics layer and is then aggregated by GBTx ASICs using e-links. The aggregated data is then transmitted over 4.8 Gb/s optical GBT links to the Common Readout Interface (CRI) boards. The CRI layer consists of 200 FPGA boards and forms an entry stage to the high-performance First-Level Event Selector (FLES) computing infrastructure where event reconstruction takes place. With this configuration, data throttling on the CRI layer is expected to prevent event corruption due to congestion of the DAQ network.
Given the number of CRI boards to serve, TFC system is organized as a hierarchical network of FPGA boards connected with bi-directional optical links. At the root of the hierarchy, there is a TFC Master that defines the time base and issues fast control commands to distribute across the network. The Master node communicates with Endpoints through one or more Submaster layers, which ensures scalability of the system. Each next downstream layer recovers clock signal from the incoming link and reuses it for further downstream communication, thus propagating the common Master clock to Endpoint nodes. Master and Submaster nodes in the network are implemented in separate FPGA boards, whereas TFC Endpoints are developed as cores that are embedded into CRI firmware.
Communication datapaths in the current prototype of the TFC system use FIFOs to handle clock domain crossings, which negatively affects the resulting quality of clock distribution. In order to overcome this limitation, it has been proposed to introduce GBT-FPGA core into the system to handle optical communication between TFC nodes. Usage of the latency-optimized version of the core is expected to significantly improve absolute data transmission latency as well as its determinism, which is crucial for the quality of clock distribution in the system.
Using the same hardware platform as TFC nodes, BNL-712, a test setup with a communication link has been implemented based on GBT-FPGA. In order to thoroughly evaluate its properties, the latency has been measured with a high-end oscilloscope controlled from a Python script via Ethernet. The current contribution presents the concept of the GBT-FPGA-based TFC system, the test setup for latency evaluation and the measurement results on performance of the proposed link.
