Speaker
Description
The data acquisition system of LHCb Upgrade I is a single stage readout followed by event building, real time reconstruction and selection. The current system already has to process 32 Tbps of data, and this will rise to above 200 Tbps with Upgrade II. The new PCIe Gen 5 readout board called PCIe400 embedding the most powerful altera’s Agilex M-series FPGA and 112 Gbps serial links is the baseline readout board for LHCb future upgrades. Presented here is the technical challenges encountered during the hardware design process with a focus on simulation performed.
Summary (500 words)
The LHCb Upgrade I implemented a triggerless data acquisition system, employing a versatile readout board performing data protocol conversion and first stage of event building. It is a key component of the chain as the same hardware is used for readout of all sub-detectors, fast command and LHC clock distribution to front-end electronics.
LHCb Upgrade II will increase the global throughput by five times. Also, the transition to 5D techniques, exemplified by the FastRICH detector, requires phase deterministic clock distribution to front-end electronics. The PCIe400 development with an output bandwidth of 400 Gbps over PCIe Gen 5 or 400GbE, and up to 48 bidirectional links at 25 Gbps, significantly outperforms current systems. Its FPGA embeds 4 million logic elements and 32 GB of memory, providing a twelvefold increase in processing capability compared to existing PCIe40 board. Latest available PLL with intrinsic jitter <100 fs RMS in combination with logic to measure and control clock phases should enable reaching ultimate precision at O(10) ps.
The design process went through meticulous studies on three major topics : Power dissipation, Power distribution and Signal Integrity. An early power consumption resulted with a total power dissipated up to 220 W which is more than 3 times what the previous generation board had been designed for. The study compared active and passive heat-sinks, solid, heat-pipe and vapor chamber heat-sink base in CFD simulations to define the parametric limits of an air cooled solution targeting our application.
PCB thickness limits imposed by the PCIe form factor also puts high constraints over the stack-up choice. Indeed, power estimation showed power rails sinking up to 100 A. Furthermore, 18 of the 22 power rails on-board are dedicated to the FPGA while power regulators can not be placed optimally all around the FPGA. This is due to placement constraints and PCB form factor. A systematic verification of voltage drop in all power planes allowed to fit a solution matching tolerances on all FPGA power supplies. A multi-thickness PCB with reduced thickness on PCIe plug zone was the solution to satisfy power distribution integrity.
Finally, 112 Gbps PAM4 striplines have 84 GHz bandwidth. Designing such striplines requires peculiar attention not to create impedance mismatch. S-parameter extraction using classic 2D simulator showed optimistic results in terms of attenuation compared to new 3D simulators considering full geometry of vias. The counterpart of 3D simulators being the processing times. We studied on how to reduce simulation time from 10 days to 10 hours. This work allowed to run comparison between various geometries especially looking at fanout, return path vias and plane openings to mitigate parasitic capacitance effect and optimize impedance matching of striplines.
In summary, the purpose of this submission is to explain the methodology, results and learning of complex board design applied to the PCIe400 case.
