21-25 May 2012
New York City, NY, USA
US/Eastern timezone

JavaFIRE: A Replica and File System for Grids

22 May 2012, 13:30
4h 45m
Rosenthal Pavilion (10th floor) (Kimmel Center)

Rosenthal Pavilion (10th floor)

Kimmel Center

Poster Distributed Processing and Analysis on Grids and Clouds (track 3) Poster Session

Speaker

Stephen Gowdy (CERN)

Description

The work is focused on the creation and validation tests of a replica and transfers system for Computational Grids inspired on the needs of the High Energy Physics (HEP). Due to the high volume of data created by the HEP experiments, an efficient file and dataset replica system may play an important role on the computing model. Data replica systems allow the creation of copies, distributed between the different storage elements on the Grid. In the HEP context, the data files are basically immutable. This eases the task of the replica system, because given sufficient local storage resources any given dataset only needs to be replicated to a particular site once. Concurrent with the advent of computational Grids, another important theme in the distributed systems area that has also seen some significant interest is that of peer-to-peer networks (p2p). P2p networks are an important and evolving mechanism that facilitates the use of distributed computing and storage resources by end users. One common technique to achieve faster file downloads from possibly overloaded storage elements over congested networks is to split the files into smaller pieces. This way, each piece can be transferred from a different replica, in parallel or not, optimizing the moments in that the network conditions are better suited to the transfer. The main tasks achieved by the system are: the creation of replicas, the development of a system for replicas transfer (RFT) and for replicas location (RLS) with a different architecture that the one provided by Globus and the development of a system for file transfer in pieces on computational grids with interfaces for several storage elements. The RLS uses a p2p overlay based on the Kademlia algorithm.

Primary author

Mr Marko Petek (UERJ)

Co-authors

Prof. Alberto Santoro (Universidade do Estado do Rio de Janeiro (BR)) Mr Claudio Fernando Resin Geyer (UFRGS) Diego Da Silva Gomes (Universidade do Estado do Rio de Janeiro (BR)) Stephen Gowdy (CERN)

Presentation Materials