Speaker
Description
The data acquisition system of LHCb Upgrade I is a single stage readout followed by event building, real time reconstruction and selection. A generic readout board, called PCIe400, embedding Altera’s flagship Agilex 7 M-series FPGA with a 4x112 Gbit/s serial interface and a PCIe Gen 5 interface constitutes a baseline for LHCb future upgrades. It also targets clock distribution with phase determinism <10 ps pk-to-pk on 46 links. Presented here are results of qualification tests performed on prototype boards.
Summary (500 words)
The LHCb Upgrade I implemented a triggerless data acquisition system, employing a versatile readout board performing data protocol conversion and the first stage of event building. It is a key component of the chain. The same hardware is used for readout of all sub-detectors, fast control commands and LHC clock distribution to front-end electronics. LHCb Upgrade II will increase the global throughput by five times. This requires the development of a new readout board, called PCIe400, with an output bandwidth of >400 Gbit/s using PCIe Gen 5, a 4x112 Gbit/s interface, and up to 48 bidirectional links at 25 Gbit/s, which significantly outperforms the current readout board: PCIe40. Its FPGA embeds 4 million logic elements, 32 GB of high bandwidth memory and transceivers capable of 116 Gbit/s bidirectional links over PAM4. Moreover, the transition to 5D techniques, exemplified by the FastRICH detector, requires phase-deterministic clock distribution. The precision requirement for Run 4 is in O(150) ps, and O(10) ps for Run 5. The card will be used for Run 4 but already embeds a full phase-deterministic time distribution system based on White Rabbit and high precision external PLLs in order to test and experiment whether the precision required for Run 5 can be reached.
During the qualification phase, we focus our study on board’s most critical features, keeping in mind quality assurance constraints for large-scale production. We prioritise self-contained tests of the board.
- Power supply ripple and transient noise compliance with on-board digital component requirements are checked. The FPGA notably specifies low noise power rails for its high-speed transceivers down to 5 mVpp. The power supply circuitry is stressed using random patterns generators emulating an applicative data processing activity. Power dissipation, evaluated at up to 200 W, is also characterised, impacting the host cooling solution.
- There are 70 bidirectional links from 1 Gbit/s to >100 Gbit/s on-board, targeting several protocols. While lower speed links can also be checked extensively using eye diagrams and jitter analysis, the high cost of 70 GHz bandwidth instruments leads us to rely on Bit Error Rate Testing (BERT) instead. We take advantage of a symmetric design looping back transmitter and receiver lanes, to measure Bit Error Rates (BER) and check the complete chain on-board.
- Initial tests of phase determinism on Agilex FPGA show that phase jumps between resets are up to 2 UI, or 200 ps at 10 Gbit/s transmission rate. We worked on a solution specific to FGT transceivers for Altera’s Agilex FPGA using transceiver internal loopback and a Digital Dual-Mixer Time Difference (DDMTD) to detect and compensate phase jumps of the serial stream against a reference clock. In order to rely on an absolute position of the phase between recompilation, a second step of the study involves freezing critical routing paths related to phase measurements. This allows evolution of the gateware logic while maintaining phase-deterministic clock distribution. We discuss the results obtained over several transceiver channels.
The results of the above tests will be presented and commented.