dCache, enabling tape systems to handle small files efficiently.

Not scheduled


1919-1 Tancha, Onna-son, Kunigami-gun Okinawa, Japan 904-0495
poster presentation Track3: Data store and access


Mr Karsten Schwank (DESY)


Increasingly, sites are using dCache to support communities that have different requirements from WLCG; as an example, DESY facilities and services now support photon sciences and biology groups. This presents new use-cases for dCache. Of particular interest is the chaotic file size distribution with a peak towards small files. This is problematic because tertiary storage systems, and tape in particular, are optimized for storing large files. Direct storing of the users files results in unacceptably poor performance. As dCache bridges the filesystem view with the underlying storage and manages transitions between media, it is the natural place to solve the poor performance from storing small files on tape. We achieved this by introducing a new service that reconciles user demand against tape behavior. The service is transparent to the users and packs files into containers (currently zip files) based on configurable policies. These containers are written directly into the same dCache, which then stores them on tape. Both the small files and the containers file then benefit from dCache features, such as caching and load-balancing. No additional storage is necessary and the service itself scales by running multiple instances within the same dCache, sharing the load. We describe the design, report on DESY's experience of running the service over the past six months, and detail the future plans for making it generally available.

Primary authors

Dr Albert Rossi (FNAL) Christian Bernardt (Deutsches Elektronen-Synchrotron (DE)) Dr Dmitry Litvintsev (FNAL) Dr Gerd Behrmann (NDGF) Mr Karsten Schwank (DESY) Dr Patrick Fuhrmann (DESY) Dr Paul Millar (Deutsches Elektronen-Synchrotron (DE)) Mr Tigran Mkrtchyan (Deutsches Elektronen-Synchrotron DESY)

Presentation materials

There are no materials yet.