Speaker
Description
In its phase-2 upgrades, all of the CMS experiment's backend electronics systems are being replaced by ATCA boards featuring AMD Xilinx UltraScale+ FPGAs and high-speed optical modules. The EMP (Extensible Modular data Processor) framework provides common infrastructural firmware components, top-level designs and associated software for multiple CMS phase-2 backend boards and systems. It implements and integrates high-speed serial link protocol engines, data capture and playback buffers, clocking components, and readout. In this talk, I will present an overview of all of the framework's key components, the associated build tooling and test suites, along with lessons learnt during the framework's development.
Summary (500 words)
In its phase-2 upgrades, all of the CMS experiment's backend electronics systems are being replaced by ATCA boards featuring AMD Xilinx UltraScale+ FPGAs and high-speed optical modules. The various phase-2 CMS backend electronics systems have different high-level purposes and I/O requirements, e.g. requiring different combinations of lpGBT links to/from detector modules, higher-speed data transfer links between off-detector boards, and readout links. The EMP (Extensible, Modular data Processor) framework consists of highly-configurable firmware designs and software libraries, designed to provide infrastructure for multiple backend boards (Apollo, BMT, and Serenity) and systems (DT, HGCal, the Level-1 Trigger, MTD, Tracker and RPC).
The EMP firmware encompasses a control bus master; generation and reception of fast, fixed-latency synchronisation signals; data transfer over high-speed serial links (lpGBT links, low-latency links between off-detector boards, and CMS readout links); and I/O data playback and capture buffers. To allow application developers to focus solely on application-specific functionality, the framework includes common top-level designs, which instantiate the individual infrastructural components and connect them to the user-created payload firmware block -- e.g. a particle/event reconstruction algorithm -- that conforms to a specific interface. Since different FPGA families have been used through the hardware development cycle, the framework has been carefully designed to support different FPGA packages and families with minimal code duplication. This required a firmware build tool which provides a high degree of customisability to select e.g. FPGA package-/family-specific source files for different target boards without duplication of common files. The build tool hides this flexibility in framework dependency files, providing users with a simple build workflow.
The EMP firmware is configured and its monitoring registers read through a single, object-oriented C++ library. Due to the large userbase and the wide range of use cases, significant attention has been paid to automated testing and validation of the firmware and software. A test suite has been developed, which configures a range of firmware components, then checks the values of monitoring registers and captured data. This is implemented such that all test cases can be run with a single command, or if desired certain combinations of test cases can be easily selected at runtime to focus testing on specific components. Comprehensive GitLab CI pipelines have been developed, including building EMP example designs with configurations that span most applications, and running the test suite on target hardware.
In this talk, I will present an overview of the design and implementation of the EMP firmware framework, along with the associated build tooling, software and test suites. I will pay particular attention to the challenges of providing this common framework for a diverse community spanning multiple boards and systems, starting from the early hardware prototyping phase. I will highlight how we addressed these challenges through e.g. the development of the comprehensive test suite and GitLab CI pipelines, and lessons that we learnt along the way for future projects.