Operational Experience Running Hadoop XRootD Fallback

13 Apr 2015, 17:15
15m
C209 (C209)

C209

C209

oral presentation Track3: Data store and access Track 3 Session

Speaker

Jeffrey Michael Dost (Univ. of California San Diego (US))

Description

In April of 2014, the UCSD T2 Center deployed hdfs-xrootd-fallback, a UCSD-developed software system that interfaces Hadoop with XRootD to increase reliability of the Hadoop file system. The hdfs-xrootd-fallback system allows a site to depend less on local file replication and more on global replication provided by the XRootD federation to ensure data redundancy. Deploying the software has allowed us to reduce Hadoop replication on a significant subset of files in our cluster, freeing hundreds of terabytes in our local storage, and to recover HDFS blocks lost due to storage degradation. An overview of the architecture of the hdfs-xrootd-fallback system will be presented, as well as details of our experience operating the service over the past year.

Primary author

Jeffrey Michael Dost (Univ. of California San Diego (US))

Co-authors

Alja Mrak Tadel (Univ. of California San Diego (US)) Frank Wuerthwein (Univ. of California San Diego (US)) Matevz Tadel (Univ. of California San Diego (US))

Presentation Materials