Operational Experience Running Hadoop XRootD Fallback

Apr 13, 2015, 5:15 PM
C209 (C209)



oral presentation Track3: Data store and access Track 3 Session


Jeffrey Michael Dost (Univ. of California San Diego (US))


In April of 2014, the UCSD T2 Center deployed hdfs-xrootd-fallback, a UCSD-developed software system that interfaces Hadoop with XRootD to increase reliability of the Hadoop file system. The hdfs-xrootd-fallback system allows a site to depend less on local file replication and more on global replication provided by the XRootD federation to ensure data redundancy. Deploying the software has allowed us to reduce Hadoop replication on a significant subset of files in our cluster, freeing hundreds of terabytes in our local storage, and to recover HDFS blocks lost due to storage degradation. An overview of the architecture of the hdfs-xrootd-fallback system will be presented, as well as details of our experience operating the service over the past year.

Primary author

Jeffrey Michael Dost (Univ. of California San Diego (US))


Alja Mrak Tadel (Univ. of California San Diego (US)) Frank Wuerthwein (Univ. of California San Diego (US)) Matevz Tadel (Univ. of California San Diego (US))

Presentation materials