September 28, 2015 to October 2, 2015
Lisbon
Europe/Zurich timezone

Pulsar IIb Design, System Integration and Next-Generation Full Mesh ATCA Backplane Test Results

Sep 30, 2015, 5:23 PM
1m
Hall of Civil Engineering (Lisbon)

Hall of Civil Engineering

Lisbon

IST (Instituto Superior Técnico ) Alameda Campus Av. Rovisco Pais, 1 1049-001 Lisboa Portugal
Poster Trigger Poster

Speaker

Jamieson Olsen (Fermilab)

Description

The Pulsar IIb is a custom ATCA full mesh enabled FPGA-based processor board which has been designed with the goal of creating a scalable architecture abundant in flexible, non-blocking, high bandwidth interconnections. The design has been motivated by silicon-based tracking trigger needs for LHC experiments. In this talk we describe the Pulsar II hardware, our full-crate integration tests, the results of our 40G and 100G ATCA full mesh backplane performance tests, and the experience gained throughout this process.

Summary

On the Pulsar IIb board a large Virtex-7 FPGA is directly connected to the backplane fabric interface. This direct connection enables the full diagnostic capabilities of the GTH serial transceivers to be used to evaluate the quality of the backplane links at speeds up to 10 Gbps.
The abundance of high speed backplane links makes the Pulsar IIb an ideal test platform for characterizing ATCA backplane performance, and to date several vendors have submitted their latest high performance 14 slot full mesh ATCA backplanes to our group for testing.

Our backplane test results indicate that channel performance consistency is one of the most significant challenges facing ATCA backplane manufacturers today. For example, one ATCA full mesh backplane rated for 40G (4 lanes x 10 Gbps) may have some channels with good wide-open eyes at 10 Gbps, but moving the Pulsar IIb board over one slot could result in marginal channels which require tuning to achieve acceptable bit error rates. Over the last year we struggled to determine if the inconsistent channel performance was caused by the Pulsar IIb PCB itself, or if it was a function of the ATCA backplane channel quality. In late 2014 COMTEL delivered to us their latest “Air/Plane” 100G full mesh ATCA backplane. This new backplane design, which uses an advanced low loss substrate material, has to date yielded the best and most consistent link performance with all Pulsar IIb prototype boards running all GTH simultaneously at 10 Gbps without error.

Transitioning from one or two Pulsar IIb boards in the shelf (backplane testing) to many boards in the shelf (system integration) posed many board management challenges that were met over the past year. First, recently we have added IPMI protocol support to the Pulsar IIb IPMC microcontroller module and verified that communication with the shelf manager board works as expected (another abstract). Secondly, we have added support for the Xilinx Virtual Cable (XVC) protocol to the IPMC microcontroller. The XVC protocol enables all FPGAs in the system to be programmed and debugged using Vivado tools over the ATCA base interface network (another abstract).

System integration testing involves not only sending data between Pulsar IIb boards over the backplane channels, but also incorporating high speed data transfers via RTM. The rear transition module (RTM) has been designed to support up to 400 Gbps bidirectional communication over fiber optic transceivers directly connected to the FPGA. We have also recently completed the design of the Pattern Recognition mezzanine card (protoPRM), which uses Xilinx Ultrascale FPGAs and attaches to standard FMC connectors on the Pulsar IIb board. The protoPRM mezzanine is designed as the core pattern recognition engine for Level-1 silicon tracking trigger R&D and supports both ASIC and FPGA based pattern recognition associative memory processors.

Primary authors

Jamieson Olsen (Fermilab) Tiehui Ted Liu (Fermi National Accelerator Lab. (US)) Zhen Hu Zijun Xu (Peking University (CN))

Presentation materials