Speaker
Description
The Embedded Monitoring Processor (EMP) is a state-of-the-art platform based on a multi-processing System-on-Chip, developed for the upgrade of the ATLAS experiment’s Detector Control System. The EMP interfaces via high-speed optical transceivers with monitoring and control functionalities of radiation-tolerant Front-Ends. Preliminary analysis revealed limitations in throughput and CPU efficiency using standard data transfer solutions between the Programmable Logic and the Processing System. Therefore, a custom DMA IP core is being developed to optimize throughput, reduce CPU load, and improve scalability. This contribution presents the modular firmware design and ongoing developments aimed at achieving a high-performance data path for slow-control data.
Summary (500 words)
The Detector Control System (DCS) of the ATLAS experiment ensures safe and coherent operation by providing a standardized interface to all subsystems and technical infrastructure, thus enabling continuous monitoring and control. To guarantee uninterrupted operation, the DCS hardware infrastructure must remain independent of the experiment’s primary power and communication systems. Moreover, components deployed within the detector cavern must be resistant to magnetic fields and possess the necessary radiation tolerance.
In the context of the High-Luminosity LHC upgrade, the existing DCS hardware no longer meets the requirements to maintain operational integrity. Therefore, higher I/O density and enhanced radiation tolerance requirements have been addressed by developing the Embedded Monitoring and Control Interface (EMCI) and the Embedded Monitoring Processor (EMP).
The EMCI serves as an interface for slow-control data exchange between various detector Front-Ends (FEs) and the EMP. It is built with CERN-developed radiation-tolerant components (lpGBT, VTRx+, and FEASTMP). The lpGBT aggregates all electrical FE signals into a single bidirectional link, which connects to the EMP via the VTRx+ optical transceiver.
The EMP is a custom-designed baseboard featuring a commercial Zynq UltraScale+ System-on-Module (SoM) on a plug-on mezzanine from Trenz, and two FireFly modules (one 12-channel transmitter and one 12-channel receiver). These 12 bidirectional optical links comply with VL+ specifications, enabling interfacing with up to 12 EMCIs in a star topology. The connection with the Supervisory Control and Data Acquisition (SCADA) system is established through a redundant dual Ethernet adapter. Furthermore, the EMP provides access to MPSoC peripherals such as GPIOs, USB3, UART, and ADC for control and monitoring.
The ongoing firmware development aims to establish a high-performance data path between the EMP and EMCI. A benchmark study of the firmware, utilising an AXI4-Lite adapter and generic Userspace I/O (UIO) driver, exposed several limitations: support for only single-burst transactions, limited data bus width, and CPU overhead due to the driver's data copying mechanism. The throughput peaked at 3 Mbps per channel when the number of links matched the number of CPU cores, resulting in complete CPU saturation. Adding more links results in a linear decrease in throughput, with all 12 links reducing it to approximately 983 kbps per link.
Although Xilinx's AXI DMA core provides high bandwidth, its driver complexity and resource overhead make it less suitable for lightweight, high-throughput applications. A DMA-based architecture effectively decouples data transfer from CPU processing, enabling high-throughput communication, significantly reducing CPU load and memory copy overhead. Therefore, a custom DMA IP core featuring an AXI4-Lite control and an AXI4 memory-mapped data interface is being developed. It is complemented by a dedicated kernel driver and a streamlined user API to achieve a scalable and efficient data transfer solution. Preliminary tests utilising a single EMP-EMCI link demonstrate throughput of up to 23 Mbps, along with a notable 25% reduction in CPU load.
This high-performance, scalable data path enhances the DCS's ability to monitor and control detector subsystems and infrastructure reliably, even as the system scales to meet increasing demands.
