Speaker
Description
The LHC phase-2 upgrades will pose unprecedent challenges in terms of timing stability to the clock delivered to thousands of nodes in an experiment. Slow phase variations could dominate the overall timing stability in a clock recovered from a high-speed optical link.
The TCLink is an integrated protocol agnostic FPGA core that can monitor and correct slow phase variations in high-speed bidirectional links. Phase-determinism (how much a clock phase changes after restarts) may play a significant role in the overall stability. In this paper we present typical phase-determinism values for TCLink implementations and potential novel architectures to improve this metric.
Summary (500 words)
The LHC phase-2 upgrades will pose unprecedent challenges to its four experiments in terms of high pile-up (number of events in the same collision). The CMS and ATLAS experiments announced the construction of dedicated timing detectors as a way of tackling this challenge. This will pose severe constraints in terms of timing stability for the timing distribution systems delivering a clock to thousands of nodes in an experiment. The overall timing stability is expected to be below 30ps rms among different nodes in a detector.
Timing distribution systems in LHC experiments typically use FPGA-based high-speed links to deliver the Bunch Clock to the front-end chips of the experiments. The impact of fast phase variations (jitter) in the clock phase are well understood and are known to be in the order of 5ps rms. Long-term variations (wander) accumulate and could play a significant role in the overall stability.
The Timing Compensated Link (TCLink) was proposed in TWEPP 2019 as a way to mitigate these slow variations related to temperature. TCLink is a protocol agnostic FPGA open soft core (https://gitlab.cern.ch/HPTD/tclink) that measures and corrects for slow phase variations in a point-to-point bidirectional high-speed optical link in a feedback control fashion. TCLink is innovative since it is fully integrated in an FPGA requiring no special external component for phase-measurement or skew adjustment. Since its first proposal, TCLink has attracted significant interest from the CMS and ATLAS experiments and it is currently being evaluated.
In addition to slow variations, phase-determinism may be an additional concern for physics results (how much a clock phase is reproducible upon a restart of a system, such as an FPGA reload between two runs). It will be demonstrated that the TCLink phase-determinism performance can vary from 1.0ps rms to 5.0ps rms for different FPGA implementations and different families (for a single hop). It will also be shown that the TCLink control loop cannot necessarily correct well for these variations due to bidirectional link symmetry reasons. Therefore, if an improvement is needed, creative fundamental architecture changes in how we implement a fixed-phase FPGA transceiver in the High-Energy physics community will be required.
While the physics performances consequences of the phase-determinism metric are still being carefully evaluated and simulated by physicists of ATLAS and CMS experiments, the TCLink team studied in detail where this variation comes from and new potential architectures to improve this performance shall it be needed.
In this paper, an overview of TCLink performance will be presented with a focus on the phase-determinism metric. An introduction to how these measurements are performed will be given together with typical values obtained in current TCLink implementations. In addition, novel architectures for having a fixed-phase in an FPGA transceiver will be discussed and the trade-offs among them (in terms of phase-determinism, resource usage and temperature stability) will be presented.
