17 September 2019
CERN
Europe/Zurich timezone

Ceph at NASA Atmosphere SIPS

17 Sept 2019, 14:00
25m
500/1-001 - Main Auditorium (CERN)

500/1-001 - Main Auditorium

CERN

46.232615, 6.045141
400
Show room on map

Speaker

Kevin Hrpcek (NASA)

Description

The NASA VIIRS Atmosphere SIPS, located at the University of Wisconsin, is responsible for assisting the Science Team in algorithm development and production of VIIRS Level-2 Cloud and Aerosol products. To facilitate algorithm development, the SIPS requires access to multiple years of satellite data occupying petabytes of space. Being able to reprocess the entire mission and provide validation results back to the Science Team in a rapid fashion is critical for algorithm development. In addition to reprocessing the SIPS is responsible for the timely delivery of near real time satellite products to NASA. To accomplish this task the Atmosphere SIPS has deployed a seven petabyte Ceph cluster employing numerous different components such as librados, EC-Pools, RBD, and CephFS. This talk will discuss choices we made to optimize the system allowing for rapid reprocessing of years of satellite data.

Primary author

Kevin Hrpcek (NASA)

Presentation materials