Speaker
Description
The CMS detector for LHC Phase-2 will be read out at 750 kHz for an event size of 7.5 MB. The optical links from detector front-ends are aggregated in ATCA back-end boards. A DAQ-and-Timing Hub (DTH) aggregates data streams from back-end boards over point-to-point links, provides buffering and transmission over 100Gb/s TCP/IP Ethernet links. The DTH is also responsible for distributing timing, control and trigger signals to the back-ends. This paper presents the requirements and the design of the DTH and the first prototype foreseen for Q4-2018. Results with Ultrascale development kits and serial HMC memories will be presented.
Summary
The CMS Detector will undergo a major upgrade for the Phase-2 of the LHC program, starting around 2026. The detector will be read out at a rate of 750 kHz by some 50k high-speed optical inks, for an event size of 7.5 MB.
The optical links from detector front-ends are aggregated in detector-dependent ATCA based back-end boards. A DAQ and Timing Hub (DTH) aggregates several data streams from multiple back-end boards over point-to-point links. These links use a custom protocol, with datagrams corresponding to one event transmitted to the DTH and flow-control in the reverse direction. The DTH combines these streams to feed high speed commercial optical links, forming the data-to-surface (D2S) network, each with 100 Gb/s or larger bandwidth. It provides buffering and transmission via TCP/IP. The D2S links carry the data to surface, connecting the DTH output via network to I/O servers for the event building. The DTH is also responsible for distributing timing, and trigger control (TTC) signals to the back-end electronics from where they are redistributed to the front-ends. The TTC signals are received by the DTH from the TCDS (Trigger Control and Distribution) master over an optical link and distributed through the backplane to all node slots. Each node slot sends its TTS status along with monitoring data to the DTH over the backplane. The DTH, in turn, elaborates a global status to generate trigger throttling as necessary.
Finally, the DTH provides monitoring of DAQ and TTC/TTS functions, and emulation of data sources, both self-triggered and externally triggered, for testing purposes.
In consideration of the structure of the ATCA crate, a standardized DTH board will be located in one of the hub slots. Because of the wide variety of required per-crate and per-board throughput, a modular design of the DTH was chosen. Data from a subset of the leaf boards in a crate can be handled by one FPGA operating independently as a DAQ unit. The baseline DTH design consists of one Timing Unit and one or more DAQ Units. In cases where the required crate throughput may exceed the maximum bandwidth of a single DTH board, it will be possible to install one or more additional DTH boards with DAQ units in other slots.
The first DTH prototype (P1), targeted for end 2018, will include a first version of one Timing Unit and one DAQ Unit. The Xilinx Kintex UltraScale+ KU15P-2 FPGA on the DAQ unit provides enough high-speed transceivers for a DAQ Unit capable of 400 Gb/s, including 24 times 16Gb/s or 16 times 25 Gb/s input pairs via FireFly mid-board optics, 16 pairs to Micron HMC (Hybrid Memory Cube) memories and 16 25Gb/s output pairs to four QFSP28 100GbE cages.
Results from ongoing work with partial DTH implementation on Ultrascale development kits and a custom add-on board with serial HMC memories will be presented, including experience with 25 Gb/s FireFly mid-board optics and transmitting TCP/IP streams from the FPGA to a 100 Gb/s Ethernet NIC in a PC-server.
