21st International Conference on Computing in High Energy and Nuclear Physics (CHEP2015)

dCache, enabling tape systems to handle small files efficiently.

Not scheduled

15m

OIST

poster presentation Track3: Data store and access

Speaker

Mr Karsten Schwank (DESY)

Description

Increasingly, sites are using dCache to support communities that have different requirements from WLCG; as an example, DESY facilities and services now support photon sciences and biology groups. This presents new use-cases for dCache. Of particular interest is the chaotic file size distribution with a peak towards small files. This is problematic because tertiary storage systems, and tape in particular, are optimized for storing large files. Direct storing of the users files results in unacceptably poor performance. As dCache bridges the filesystem view with the underlying storage and manages transitions between media, it is the natural place to solve the poor performance from storing small files on tape. We achieved this by introducing a new service that reconciles user demand against tape behavior. The service is transparent to the users and packs files into containers (currently zip files) based on configurable policies. These containers are written directly into the same dCache, which then stores them on tape. Both the small files and the containers file then benefit from dCache features, such as caching and load-balancing. No additional storage is necessary and the service itself scales by running multiple instances within the same dCache, sharing the load. We describe the design, report on DESY's experience of running the service over the past six months, and detail the future plans for making it generally available.