14-18 October 2013
Amsterdam, Beurs van Berlage
Europe/Amsterdam timezone

Tool for Monitoring and Analysis of Large-Scale Data Movement in (Near) Real Time

17 Oct 2013, 11:23
Veilingzaal (Amsterdam, Beurs van Berlage)


Amsterdam, Beurs van Berlage

Oral presentation to parallel session Facilities, Production Infrastructures, Networking and Collaborative Tools Facilities, Infrastructures, Networking and Collaborative Tools


Dr Wenji Wu (Fermi National Accelerator Laboratory)


Fermilab is the US-CMS Tier-1 Centre, as well as the main data centre for several other large-scale research collaborations. As a consequence, there is a continual need to monitor and analyse large-scale data movement between Fermilab and collaboration sites for a variety of purposes, including network capacity planning and performance troubleshooting. To meet this need, Fermilab designed and implemented a network traffic characterization system for our large-scale bulk data transfers. The original version of the system simply analysed flow data sequentially on a conventional multicore system. That design had two significant limitations. First, there was an appreciable delay in the analysis. While the results were still useful for network characterization studies and capacity planning purposes, they were not available in near real-time. Such a capability would open up more operationally-oriented uses. Second, the system would not scale well to 40GE and 100GE network environments, due to the sequential analysis. Fermilab is currently developing an enhanced system which remedies those limitations. The system consists of two major components, a flow-data collection component and a data analysis engine based on GPU (graphic processing unit) technology. Our system exploits the data parallelism that exists within network flow data to provide accurate monitoring and near real-time analysis of bulk data movement. The system is designed to work in 40GE/100GE network environments. In this paper, we discuss the architecture and design of our system, including some of the applications available. Results from analysis on production traffic will be incorporated into the presentation.

Primary author

Dr Wenji Wu (Fermi National Accelerator Laboratory)


Mr Phil Demar (Fermilab)

Presentation Materials