Speaker
Description
The CMS Phase-2 upgrade of the CMS Data Acquisition (DAQ) system is based on the DAQ and Timing Hub (DTH) custom board. The DTH serves as an interface, connecting sub-detector front-end electronics to the DAQ, as well as timing and trigger control and distribution system. This work focuses on the DAQ functionality of the DTH, detailing the current status of the project, including the DAQ firmware and control software. First integration with Phase-2 sub-detectors and future plans are presented.
Summary (500 words)
The CMS experiment at the LHC is preparing for the high-luminosity era, requiring a major upgrade of the Data Acquisition (DAQ) system to cope with the significantly increased data volumes. Central to this upgrade are two custom-designed Advanced Telecommunications Computing Architecture (ATCA) boards: the DAQ and Timing Hub (DTH400) and the DAQ800.
The DTH400 is a dual-FPGA board that connects sub-detector back-end boards to the DAQ, as well as timing and trigger control and distribution system. It features two Xilinx Virtex UltraScale+ VU35P FPGAs, specifically chosen for it’s large High Bandwidth Memory (HBM). One FPGA handles trigger control and trigger and timing distribution, while the other performs high-speed data aggregation from six 4-channel optical engines (Firefly) operating at 25 Gbit/s, providing a total throughput of 400 Gbit/s and converting data from a custom high speed serial protocol (SLinkRocket) to 100 Gbit TCP/IP. The HBM provides large TCP/IP socket-buffers at the required high throughput. To meet the different throughput requirements of CMS sub-detectors, a complementary board, the DAQ800, was developed. The DAQ800 integrates two DAQ units equivalent to the DAQ unit of the DTH400 but without timing and trigger functionality, providing 800 Gbit/s throughput. The DAQ unit firmware allows flexible routing of the TCP-streams (one per incoming SlinkRocket) to 5 100Gbps QSFP28 optical tranceivers, allowing for a flexible distribution of the data on the available output links and for optimising the available network resources. After successful prototyping and pre-production, full production of the DTH board is now ongoing.
A Rear Transition Module (RTM) in the DTH/DAQ800 system uses a Xilinx Kria K26 System-on-Module (SoM) with a Zynq UltraScale+ MPSoC for embedded control and monitoring. The control application is based on a modular and efficient software stack, using a C++/Python hardware abstraction layer, a RESTful HTTP server for control, and an MQTT client for monitoring. The control application is containerised and managed in a Kubernetes cluster. Gitlab CI/CD is used for testing and deployment during development.
A test integration facility has been set up at CERN in order to validate the integration of Phase-2 sub-detectors with the central TCDS/DAQ. It will allow sub-detectors to integrate their systems with the central Trigger and DAQ. The platform validates the real-time data flow and trigger synchronization using the final hardware and software components of the Phase-2 DAQ. This facility allows integration work to proceed while the infrastructure of the CMS experiment is upgraded and hence not operational.