Speaker
Description
Summary
This paper is devoted to the challenges faced and lessons learnt during the development and commissioning of the GCT system and refer to the architecture of the design and the implementations of high speed serial links. Both are likely to be used in future systems and are of value to the larger LHC trigger community.
The main challenge with the GCT and with most trigger systems is the high bandwidth requirements coupled with the fact that data often needs to be shared or duplicated, and done so with low latency. The GCT uses a mixture of high speed serial links and wide parallel busses. The high speed serial links are necessary to concentrate the data into a single FPGA, thus reducing data sharing requirements and making the processing efficient. The latency cost of these links is not negligible and thus wide parallel busses operating conservatively at 80MHz are used for the rest of the system.
The GCT is modular, which allowed multiple design teams to work in parallel in the initial stages of the product. It also simplified each board, thus reducing the layout and design time. It has also allowed the GCT-to-GT links to be replaced without requiring complex changes to the main 9U VME data processing card.
The revised GCT was originally designed to accommodate the existing interfaces to the GT and RCT:
The original GCT-to-GT interface was based on National Semiconductor DS92LV16 electrical high speed serial links operating just beyond specification. In the revised GCT design these links were placed on a dual CMC daughter card. The interface was successfully tested. However, when new shorter cables were used for the GCT-to-GT links it was noticed that the SERDES links occasionally lost lock. This was traced to reflections from the receiver. A new interface is being built based on the Virtex 5 FPGA and 16 bidirectional optical links operating up to 3.2Gb/s.
The GCT input interface with the Regional Calorimeter Trigger (RCT) consists of 63 Source Cards which transmit the RCT data using high speed optical links to the 8 Leaf Cards of the GCT. The source cards convert the parallel differential ECL from the RCT to high speed serial optical signals with 8B/10B encoding and a CRC check per orbit. During commissioning it was noted that occasionally one of the links on the Leaf card was generating CRC errors. The original hypothesis was that this was due to the quality of the clock used to drive the SERDES signals, which was close to the specification limit. The links were therefore modified to use low jitter local oscillators and operate asynchronously to the TTC clock at 2.0Gb/s (data rate = 1.28Gb/s). However, this did not solve the problem and the fault was eventually traced to firmware tools incorrectly placing the design despite constraints to the contrary. The system continues to operate in this way which has the benefit that we can use a very low jitter clock source and the latency is not affected because the increase in latency due to the clock domain bridge on the Source card is cancelled by the internal logic in the SERDES units operating faster.
