Speaker
Description
Accurate clock and time distribution is a key requirement for self-triggered streaming data acquisition in the CBM experiment. This distribution is handled by the Timing and Fast Control (TFC) system by clock forwarding and broadcasting the common time over latency-deterministic optical links in a hierarchical FPGA network. The point-to-point optical connections are served by the latency-optimized GBT-FPGA core, which has been developed at CERN. In the presented work, the performance of GBT-FPGA links for time and clock distribution in a scaled TFC system with multiple hops and endpoints has been investigated.
Summary (500 words)
The future Compressed Baryonic Matter (CBM) experiment aims to study strongly interacting matter at high baryonic densities by measuring rare diagnostic probes at interaction rates of up to 10^7 events per second. Due to the complexity of the potential trigger signatures, this approach requires at least partial event reconstruction to select events of interest. To facilitate this, the CBM experiment will be equipped with a streaming data acquisition (DAQ) system with self-triggered front-end electronics (FEE). Event reconstruction and selection is time-based and will be completely performed online by software running in a computing farm.
Up to 1 TB/s of timestamped experimental data is produced by the FEE and concentrated into optical links using the readout boards (ROB) based on the GBTx chipset. The 4.8 Gb/s optical links transfer the data to a layer of about 200 PCIe-based Common Readout Interface (CRI) boards. The CRI boards are equipped with an FPGA that reformats the data before passing it on to the First-level Event Selector (FLES) cluster for event reconstruction. This readout scheme requires that a common clock signal, together with the time information for accurate and coordinated timestamping, are provided by the CRIs to the FEE via the ROBs. This is achieved by leveraging deterministic downstream clock and data transport capability of GBTx ASICs. The time and clock in the CRI boards, however, must be synchronized independently, which is handled by the Timing and Fast Control (TFC) system. Here, the clock phase stability requirements of the FEE define the requirements for the distribution of common clock and time.
To accomplish this task, the TFC system takes the approach of forwarding the clock and time messages from a central master node over an optical network of FPGA boards to the individual CRIs. The challenge of this approach is to ensure sufficient determinism of both phase of the forwarded clock and the time message latency in a tree network, without interfering with the latency-critical Fast Control messages. The relative clock and time offset in all endpoints must be stable to a maximum of 200 ps at all times, including full system restarts.
Implementation of the proposed approach requires the optical point-to-point connections in the hierarchical network to be deterministic in the downstream direction (TFC master to CRIs). Here, the GBT-FPGA core has been selected, as it features latency-deterministic datapaths by design. Whereas latency determinism of a direct GBT-FPGA connection between two FPGA devices has been verified, it requires further study to characterize the downstream link latency determinism of the entire TFC network. With the goal of generating sufficient insight to estimate the quality of clock and time distribution in the final experimental setup, the current work focuses on characterizing latency variance over multiple hops, as well as relative time error between multiple endpoints.
