14-18 October 2013
Amsterdam, Beurs van Berlage
Europe/Amsterdam timezone

Experience of a low-maintenance distributed data management system

17 Oct 2013, 14:36
20m
Keurzaal (Amsterdam, Beurs van Berlage)

Keurzaal

Amsterdam, Beurs van Berlage

Oral presentation to parallel session Distributed Processing and Data Handling B: Experiment Data Processing, Data Handling and Computing Models Distributed Processing and Data Handling B: Experiment Data Processing, Data Handling and Computing Models

Speaker

Mr Wataru Takase (High Energy Accelerator Research Organization (KEK), Japan)

Description

In this paper we report on the setup, deployment and operation of a low-maintenance, policy-driven distributed data management system for scientific data based on the integrated Rule Oriented Data System (iRODS). The system is located at KEK, Tsukuba, Japan with a satellite system at QMUL, London, UK. The system has been running stably in production for more than two years with minimal management overhead. We also report on the experience gained and lessons learnt during the setup and operation. The management tools that were developed to support the production system are also described. In addition we describe a simple XOR-based approach to file replication that reduces the amount of storage space consumed. In situations of large data volumes this approach can be of great benefit.

Primary authors

Dr Adil Hasan (University of Liverpool, UK) Prof. Francesca Di Lodovico (Queen Mary College, University of London, UK) Mr Wataru Takase (High Energy Accelerator Research Organization (KEK), Japan) Ms Yoshimi Matsumoto (High Energy Accelerator Research Organization (KEK), Japan)

Co-authors

Prof. Takashi Sasaki (High Energy Accelerator Research Organization (KEK)) Prof. Yoshiyuki Watase (High Energy Accelerator Research Organization (KEK))

Presentation Materials