Sep 22 – 26, 2014
Centre des Congrès - Aix en Provence, France
Europe/Zurich timezone

Architectural Improvements and Technological Enhancements for the APEnet+ Interconnect System

Sep 24, 2014, 4:46 PM
1m
Centre des Congrès - Aix en Provence, France

Centre des Congrès - Aix en Provence, France

14 boulevard Carnot 13100
Poster Opto Second Poster Session

Speaker

Andrea Biagioni (INFN Rome Section)

Description

In this paper we describe the latest generation of APEnet+ network interface card. This new APEnet+ generation delivers a point-to-point, low-latency, 3D-torus NIC integrated in a PCIe Gen3 board based on a state-of-the-art, 28nm Altera Stratix V FPGA. The NIC features a network architecture designed following the Remote DMA paradigm and tailored to tightly bind the computing power of modern GPUs to the communication fabric. For the new APEnet+ card we show characterizing figures as achieved bandwidths, eye diagrams and BER obtained by exploiting new high performance ALTERA transceivers and PCIe Gen3 compliancy.

Summary

The APEnet+ project delivered a point-to-point PCIe Gen2 interconnect adapter to be employed in hybrid x86+GPU computing clusters with a 3D toroidal network mesh. APEnet+ high performance and low latency capabilites are the result of a network architecture designed following the Remote DMA paradigm which is tailored to tightly bind the computing power of modern GPUs to the communication fabric. Doing so on the APEnet+ board was possible by employing latest standards for the physical interconnect - SFF-8436 with its QSFP+ modules - and exploit them by means of the vast resources provided by a state-of-the-art FPGA platform like the Stratix IV the board is based upon.
Following up with enhancements in these areas - the most significant being an upgraded interconnect standard (SFF-8665) with new zQSFP plugs and the latest generation for the PCIe standard - means employing an evolved FPGA platform able to integrate and exploit these improvements alongside added functionality. This requires a redesign of APEnet+ to support the new PCIe Gen3 protocol, faster transceivers and a much larger amount of programmable resources.
The chosen platform for this redesign exploration is the Stratix V FPGA Development Kit. The board is based on an 28nm technology FPGA (5SGXEA7K2F40C2N) and sports a PCIe x8 connector, two HSMC connectors for exporting high bandwidth links over a daughterboard, a QSFP connector with optical cage and an Ethernet PHY 10/100/1000Mbps copper connector.
A PLDA-proprietary component - QuickPCIe Expert - was chosen as a Gen3-compliant core interface towards the PCIe bus. With this core, either an Avalon or an AXI bus can be selected as user interface; the choice of implementing the latter gives us room for preparing the new APEnet+ environment to deal with the impact brought by a full-featured ARM processor which is slated to be one of the most significant additions on FPGA platforms of the next generation.
The new PLDA core also allows for greater freedom when choosing among different strategies for the DMA channels management.
In this work we will show how performances vary as a result of different implementations in this regard; moreover, we will show updated, characterizing figures for APEnet+ over the new high performance ALTERA transceivers provided by the Stratix V Development Kit as achieved bandwidths, eye diagrams and BER.

Primary authors

Alessandro Lonardo (INFN Rome Section) Andrea Biagioni (INFN Rome Section) Davide Rossetti (Nvidia Corporation) Elena Pastorelli (INFN Rome Section) Francesca Lo Cicero (INFN Rome Section) Francesco Simula (INFN Rome Section) Frezza Ottorino (INFN Rome Section) Laura Tosoratto (INFN Rome Section) Michele Martinelli (INFN Rome Section) Pier Stanislao Paolucci (INFN Rome Section) Piero Vicini (INFN Rome Section) Roberto Ammendola (INFN)

Presentation materials