21-25 May 2012
New York City, NY, USA
US/Eastern timezone

APEnet+: a 3-D Torus network optimized for GPU-based HPC Systems

22 May 2012, 13:30
4h 45m
Rosenthal Pavilion (10th floor) (Kimmel Center)

Rosenthal Pavilion (10th floor)

Kimmel Center

Poster Computer Facilities, Production Grids and Networking (track 4) Poster Session


Laura Tosoratto (INFN)


The emerging of hybrid GPU-accelerated clusters in the supercomputing landscape is a matter of fact. In this framework we proposed a new INFN initiative, the QUonG project, aiming to deploy a high performance computing system dedicated to scientific computations leveraging on commodity multi-core processors coupled with last generation GPUs. The multi-node interconnection system is based on a point-to-point, high performance, low latency 3-d torus network built in the framework of the APEnet+ project: it consists of an FPGA-based PCI Express board exposing six full bidirectional links running at 34 Gbps each, and implementing RDMA protocol. In order to enable significant access latency reduction for inter-node data transfer a direct network-to-GPU interface was built. The specialized hardware blocks, integrated in the APEnet+ board, provide support for GPU-initiated communications using the so called PCI Express peer-to-peer (P2P) transactions. To this end we are strongly collaborating with NVidia GPU vendor. The final shape of a complete QUonG deployment is an assembly of standard 42U racks, each one capable of ~80 TFlops/rack of peak performance, at a cost of 5 KEuro/TFlops and for an estimated power consumption of 25 KW/rack. A first reduced QUonG system prototype is expected to be delivered by the end of the year 2011. In this talk we will report on the status of final rack deployment and on the 2012 R&D activities that will focus on performance enhancing of the APEnet+ hardware through the adoption of new generation 28nm FPGA allowing the implementation of PCI-e Gen3 host interface and the addition of new fault tolerance oriented capabilities.

Primary author

Dr Piero Vicini (INFN Roma - Roma)


Dr Alessandro Lonardo (INFN Roma - Roma) Dr Davide Rossetti (INFN - Rome) Dr Francesca Lo Cicero (INFN Roma - Roma) Dr Francesco Simula (Sapienza Universita' di Roma) Laura Tosoratto (INFN) Dr Pier S. Paolucci (INFN Roma - Roma) Dr Roberto Ammendola (INFN Tor Vergata - Roma) Dr andrea biagioni (INFN Roma - Roma) Dr ottorino Frezza (INFN Roma - Roma)

Presentation Materials