Sharing lattice QCD data over a widely distributed file system

Not scheduled
15m
OIST

OIST

1919-1 Tancha, Onna-son, Kunigami-gun Okinawa, Japan 904-0495
poster presentation Track3: Data store and access

Speaker

Tomoteru Yoshie (University of Tsukuba)

Description

JLDG is a data-grid for the lattice QCD (LQCD) community in Japan. Several large research groups in Japan have been working on lattice QCD simulations using supercomputers distributed over distant sites. The JLDG provides such collaborations with an efficient method of data management and sharing. File servers installed on 9 sites are connected to the NII SINET VPN called HEPnet-J/sc and are bound into a single file system with the GFarm, a grid-base file system software. Because the file system looks the same from any sites, users can do analyses (measurement of physical quantities) on a supercomputer on a site, using data generated and stored in the JLDG at a different site. Since the official start of operation in 2008, the JLDG has been improved in various ways. Among others, the following two have drastically improved usability of the JLDG. 1) Implementation of FUSE mount which enables users to mount the JLDG file system as a unix file system. 2) Development of fast data copy sub-system between the JLDG and the HPCI Shared Storage, where the HPCI is a Japanese national project started in 2011 for constructing High Performance Computing Infrastructure. We present a brief description of hardware and software of the JLDG, focusing mainly on the above two improvements and report performance and statistics of the JLDG. As of October 2014, 11 research groups (66 users) store their daily research data of 4.0PB including replica and 63 million files in total. Number of publications for works used the JLDG is 105. The large number of publications and recent rapid increase of disk usage convince us that the JLDG has grown up into a useful infrastructure for LQCD community in Japan.

Primary authors

Osamu Tatebe (University of Tsukuba) Tomoteru Yoshie (University of Tsukuba) Toshiyuki Amagasa (University of Tsukuba)

Co-authors

Akira Ukawa (RIKEN Advanced Institute for Computational Science) Hideo Matsufuru (High Energy Accelerator Research Organization (KEK)) Hideyuki Jitsumoto (Tokyo Institute of Technology) Hiroaki Togawa (Osaka University) Hiroyuki Kamano (Osaka University) Kazumi Fukumura (Kyoto University) Ken-Ichi Ishikawa (Hiroshima University) Kohtaroh Miura (Nagoya University) Mitsuhisa Sato (University of Tsukuba) Naoya Ukita (University of Tsukuba) Noriyoshi Ishii (Osaka University) Shinji Takeda (Kanazawa university) Sinya Aoki (University of Tsukuba) Takeshi Yamazaki (University of Tsukuba) Takumi Doi (RIKEN) Tatsumi Aoyama (Nagoya University) Yasumichi Aoki (Nagoya University) Yasushi Watanabe (RIKEN) Yoshiaki Mikami (Hitachi Solutions East Japan, Ltd.) Yukiko Konno (Hitachi Solutions East Japan,Ltd.)

Presentation Materials