Speaker
Description
New detectors for High Energy Physics are being developed to withstand the high-luminosity upgrade of the LHC. This upgrade increases the requirements for phase stability of the timing distribution systems (<10ps RMS). Commercial transceivers do not meet these stringent requirements, although ad-hoc solutions have been implemented to measure phase drifts and compensate, these are FPGA-specific and limited to specific paths.
This paper presents the design and results of the Phase Stable Link (PSLink) mezzanine, an FPGA-agnostic phase measurement and compensation system, which allows to stabilize the entire cascade of multiple FPGAs, reaching a phase stability of 1ps RMS per hop.
Summary (500 words)
With the High-Lumi upgrade of the LHC the number of collision events per bunch crossing increases from the current 60 to at least 140 (pile-up), requiring detectors with precise timing resolution to disentangle the collisions in a time window of 325ps. For example, the High Granularity Timing Detector (HGTD) has a resolution of 30ps, requiring a stability of less than 10ps RMS for the entire timing distribution system.
Timing distribution systems are generally composed by a cascade of FPGAs that receive and retransmit timing signals embedded in serial streams to multiple endpoints on the detectors.
FPGA transceivers have shown inconsistencies in terms of phase stability after resets or power cycles. Optical fibers and FPGA logic are also highly sensitive to temperature variations (of the order of 15 to 30ps per degree C for a 100m fiber, and between 2 and 4ps per degree C for FPGA logic).
Several detailed studies have been carried out in the community in recent years to overcome these difficulties. Some have been presented at TWEPP, such as the TCLink for stabilizing the operation of transmitters in AMD/Xilinx FPGAs of the Ultrascale/Ultrascale+ family at ±1.5ps peak-to-peak, or compensating for phase variations due to temperature changes in optical fibers. Other techniques for constraining the receivers of these FPGAs have also been studied, with promising results of the order of ±5 to 10ps peak-to-peak per board. All solutions are using internal FPGA features like transceiver phase interpolators for phase shifting, and Digital Dual Mixer Time Difference (DDMTD) logic for phase variation measurement.
Although showing promising performance, these solutions remain intrinsically linked to the architecture and specificities of FPGA families and their transceivers. These solutions also suffer from imperfections in the digital phase measurements due to routing mismatches and non-linearities.
Furthermore there is no solution that covers the entire path of the timing distribution system.
To overcome these drawbacks, a study carried out as part of the CERN EP RnD and ECFA DRD7 programs, proposes to implement a generic solution to compensate for the instability of FPGAs, irrespective to their performance in terms of phase stability and temperature sensitivity.
The paper presents an optimized implementation on hardware, a mezzanine card, of the concept presented at TWEPP24.
The PSLink performs phase measurements between the received and re-transmitted data streams on the FPGA and compensates with phase shifters, to maintain the delay stable.
The original concept is extended to compensate for phase variations not only on the FPGA path, but also on the fibers connecting the FPGAs.
A novel DDMTD-based solution for performing phase measurements directly on data streams is adopted, allowing for a significant cost, space and bandwidth-limits reduction by removing the need for the Clock and Data Recovery (CDR) chips.
The performances of two versions of the design are compared, one adopting the direct measurement on data stream, another following the original concept of clock recovery. Both versions reached the targeted phase stability of 1ps RMS per hop under perturbations such as temperature drifts and transceiver restarts.