Speaker
Description
With the streaming data acquisition scheme planned for the CBM experiment, the quality of event reconstruction depends on the accuracy of clock and time distribution. The Timing and Fast Control (TFC) system ensures that all the readout FPGA boards are aligned in time on the sub-clock and the absolute scales. This is achieved by distributing the time information over an optical link with deterministic latency and clock recovery. In this work, implementation of reliable deterministic time forwarding has been explored. This allows for a scalable multi-hop architecture, to handle the large number of FPGA boards planned in the experiment.
Summary (500 words)
Aimed at studying strongly interacting matter at high baryonic densities at interaction rates of up to 10^7 events per second, the Compressed Baryonic Matter (CBM) experiment faces the challenge of high complexity in the potential trigger signatures, making the implementation of a hardware trigger practically impossible. To tackle this, the CBM experiment is being developed with a streaming data acquisition system. With this approach, all data collected by the self-triggered front-end electronics (FEE) are forwarded via the PCIe-based Common Readout Interface (CRI) FPGA boards to a computer farm, where high-performance software selects events of interest based on time-based reconstruction. The time information is obtained from the timestamps assigned to each hit message by the FEE.
Synchronization of the FEE with sub-nanosecond precision is essential for accurate hit timestamping and, consequently, for efficient event reconstruction and selection. Achieving this synchronization requires distribution of both the clock signal and a common time base from a central timing system to the entire experiment. In CBM, the task of distributing the 40 MHz global clock and time to the CRI boards is handled by the Timing and Fast Control (TFC) system, with the CRI boards propagating the timing to their corresponding FEE clusters.
The TFC system is based on the FPGA boards FLX-712, interconnected with bidirectional optical links in a hierarchical topology. The global time reference is defined in the master node and is distributed downstream with a deterministic latency to about 200 CRI endpoints, sharing the links with fast control messages. The clock signal is embedded in the data link and is propagated through clock recovery and its subsequent retransmission. Since each of the FLX-712 boards has 48 optical interfaces, an intermediate layer of submaster nodes is required to serve all endpoints in the experiment.
Even with a sufficiently deterministic link between two TFC nodes, the presence of intermediate nodes inevitably impacts the overall latency determinism in the system. With a focus on the implementation of the submaster nodes, this work investigates the sources of latency variation associated with using a large number of parallel downstream links and multiple clock domains in the path of forwarded data, and the methods to minimize their effects. Eventually, implementation of the submaster node is evaluated in a minimal version of a TFC system with 2 hops.
