22–26 Sept 2014
Centre des Congrès - Aix en Provence, France
Europe/Zurich timezone

The C-RORC PCIe Card and its Application in the ALICE and ATLAS Experiments

25 Sept 2014, 11:10
25m
Centre des Congrès - Aix en Provence, France

Centre des Congrès - Aix en Provence, France

14 boulevard Carnot 13100
Oral Logic Logic & Packaging

Speaker

Heiko Engel (Johann-Wolfgang-Goethe Univ. (DE))

Description

The ALICE and ATLAS DAQ systems read out detector data via point-to-point serial links into custom hardware modules, the ALICE RORC and ATLAS ROBIN. To meet the increase in operational requirements both experiments are replacing their respective modules with a new common module, the C-RORC. This card, developed by ALICE, implements a PCIe Gen 2 x8 interface and supports twelve optical links via three QSFP transceivers. This paper presents the design of the C-RORC, its performance and its applications in the ALICE and ATLAS experiments.

Summary

The LHC collaborations have performed significant development to consolidate their electronics and online systems for Run 2. These developments aim to efficiently use the LHC luminosity anticipated for the period 2015-2018 and include an increase of the bandwidth available in the online systems.

Two LHC experiments, ALICE and ATLAS, will use the same new card, the ALICE Common Read-Out Receiver Card (C-RORC), configured with experiment-specific firmware, to interface up to twelve digital optical links to the input/output bus (PCIe Gen 2) of the online computers. The ATLAS specific firmware is known as the ROBINNP firmware.

All the information sent by the 18 sub-detectors composing ALICE are currently read out by the data acquisition system using ~ 500 optical links called DDL (Detector Data Link), each one with a maximum throughput of 2.1 Gb/s. ALICE will face a partial upgrade in 2015 to cope with the increased luminosity of the LHC. Two detectors (the Time Projection Chamber and the Transition Radiation Detector) will consolidate their front-end electronics to increase the acquisition rate to 4 and 5 Gb/s respectively. For this reason a new common DAQ and HLT (High-Level Trigger) readout card has been developed with a higher input link capability - 12 links (using three QSFP transceivers) instead of 2 - and a faster PCIe interface (an eight-lane PCIe Gen 2 instead of Gen 1). The protocol used to send data is still the same but the links run at a higher speed, up to a maximum of 6 Gb/s per link.

The off-detector read-out chain of the ATLAS experiment, consists of sub-detector specific Read-Out Drivers (RODs), which output event fragments to the Read-Out System (ROS) via point-to-point optical Read-Out Links (ROLs). In Run 1 the ROS was implemented with 4U high rack-mountable server-class PCs, each equipped with up to four custom 64-bit PCI cards, the ROBINs. For Run 2 the ROS will use 2U PCs, each with two C-RORCs running ROBINNP firmware providing the ROBIN functionality. The firmware currently implements a PCIe x8 Gen 1 interface, which supports read-out of more than the required 50% of all data input on 12 ROLs running at 200 MB/s. After an upgrade to Gen 2 100% read-out could be possible.

A single Xilinx Virtex-6 series FPGA handles data input from the 12 optical links and buffering in the on-board DDR3 memory. It is also capable of doing on-the-fly data processing and initiating DMA transfer of event data directly from the links or from the on-board memory to the PC memory. The common use of the same complex hardware demonstrates the flexibility of programmable logic to implement different protocols satisfying the different requirements of two experiments.

Author

Heiko Engel (Johann-Wolfgang-Goethe Univ. (DE))

Co-authors

Andrea Borga (NIKHEF (NL)) Andreas Kugel (University of Heidelberg) Barry Green (RHUL - Royal Holloway, University of London) Csaba Soos (CERN) David Francis (CERN) Filippo Costa (CERN) Fred Wickens (STFC - Rutherford Appleton Lab. (GB)) Gordon Crone (University College London (UK)) Joern Schumacher (CERN / University of Paderborn) Joseph Vermeulen (NIKHEF (NL)) Louis Tremblet (CERN) Markus Joos (CERN) Prof. Pedro Teixeira-Dias (RHUL - Royal Holloway, University of London) Per Werner (CERN) Mr Pierre Vande Vyvre (CERN) Tivadar Kiss (Hungarian Academy of Sciences (HU)) Udo Wolfgang Kebschull (Johann-Wolfgang-Goethe Univ. (DE)) Wainer Vandelli (CERN) Mr William Panduro Vazquez (RHUL - Royal Holloway, University of London)

Presentation materials