Dr Iosif Legrand (CALTECH) Ramiro Voicu (CALTECH)
The efficient use of high-speed networks to transfer large data sets is an essential component for many scientific applications including CERN’s LCG experiments. We present an efficient data transfer application, Fast Data Transfer (FDT), and a distributed agent system (LISA) able to monitor, configure, control and globally coordinate complex, large scale data transfers. FDT is an Application for Efficient Data Transfers which is capable of reading and writing at disk speed over wide area networks (with standard TCP). If it is used for memory to memory transfers between two computers it can saturate a 10Gb/s WAN link. Disk to disk transfers between several servers can saturate a 10Gb/s in both directions. It is written in Java, runs an all major platforms and it is easy to use. FDT is based on an asynchronous, flexible multithreaded architecture able to balance and optimize the access to disks and to control the flow of data in the network through multiple streams. It streams datasets continuously, using a managed pool of buffers through one several TCP sockets in parallel and is using independent threads to read and write on each physical device. LISA (The Localhost Information Service Agent) is lightweight dynamic services that provides complete system monitoring and is capable to dynamically configure the system or running applications. It provides the functionality to orchestrate and optimize distributed large data transfers at the Supercomputing 2006 Bandwidth Challenge between more than 200 systems around the world.