Speaker
Description
FELIX, the PCIe based framework has been used in the DAQ system of ATLAS Phase-I upgrade and the APA (Anode Plane Assemblies) readout in Single-Phase ProtoDUNE experiment. For the ATLAS HL-LHC upgrade, the fiber optical links from front-ends will have higher speed. This manuscript introduces a FELIX demonstrator board with PCIe interface designed for the HL-LHC upgrade. In this board, 25+ Gbps optical links will be supported with both optical transceivers and FMC+ mezzanine, in addition to FPGA embedded PCIe hard IP block for high bandwidth interface with the CPU.
Summary
The Front-End LInk eXchange (FELIX) is a novel readout system to interface with front-end
electronics and a commodity network in the ATLAS Phase-I upgrade. It has also been used in the neutrino experiment Single-Phase ProtoDUNE. The Phase-I FELIX card has 4 pairs of 12-channel MiniPODs which support 48 bidirectional fiber optical links with speed up to 14Gbps. The interface to the PC is a 16-lane Gen-3 PCIe supporting a throughput up to 101Gb/s.
For the HL-LHC upgrade, the event rate will increase from 100 kHz to about 1 MHz. Meanwhile the event size will increase from ~2 MB to ~6 MB. After the upgrade, the speed of lpGBT links (10.24Gbps) will be about twice of current GBT links (4.8Gbps). The custom lightweight protocol, the FULL mode will use higher link speed than the one (9.6Gbps) implemented for Phase-I upgrade. Since the HL-LHC upgrade will be in 2024, the new generation PCIe Gen-4 or Gen-5 will be considered for the baseline FELIX design. To evaluate various key technologies for the FELIX card in HL-LHC, a demonstrator board is developed. The Xilinx FPGA VU9P is used which has Gen-3/Gen-4 PCIe hard IP blocks. 64 high-speed optical links will be supported by using different modules like the BOA from Finisar, the OBT from Amphenol and the FireFly from Samtec. These optical modules support link speed up to 25 or 28 Gbps. Verification of these modules have been done with a Xilinx evaluation board VCU108 in the lab. The designed demonstrator PCIe card will also be used in the evaluation of DUNE readout, which has one candidate solution to transfer the received streaming APA data to a mezzanine for further data processing. In this design, the FMC+ mezzanine interface is implemented. Besides the data bus with 34 pairs of differential lines, 24 high speed GTY transceivers are also connected from the FELIX FPGA to the FMC+ mezzanine. A timing mezzanine card will be used on this PCIe card to support different timing systems. Four differential clock signals and more than 10 differential data signals will be connected between the FELIX FPGA and timing mezzanine. The board will have a 288-pin DDR4 DIMM slot to support high bandwidth data buffering.
To support the complex design of the FELIX demonstrator, a new 24-layer stack-up will be used which has a thickness of about 2.67mm. To make the PCIe edge connector be compatible with the PCIe specification, the connector area will only have 14 layers for thickness of 1.6mm. Two sub-assemblies will be applied to layers 1-14, and layers 15-24, and sequential lamination will be required for the PCB fabrication. To improve the signal integrity for the high-speed links connected to the GTY transceivers, stubs of these traces should be as short as possible. Several types of back-drills are used for vias on these high-speed traces.
This manuscript will introduce the design considerations and technical implementations of this PCIe card. Preliminary test results will also be presented.
