Nov 4 – 8, 2019
Adelaide Convention Centre
Australia/Adelaide timezone

Distributed Caching in the WLCG

Nov 5, 2019, 3:30 PM
Hall F (Adelaide Convention Centre)

Hall F

Adelaide Convention Centre

Poster Track 4 – Data Organisation, Management and Access Posters


Max Fischer (Karlsruhe Institute of Technology)


With the evolution of the WLCG towards opportunistic resource usage and cross-site data access, new challenges for data analysis have emerged in recent years. To enable performant data access without relying on static data locality, distributed caching aims at providing data locality dynamically. Recent work successfully employs various approaches for effective and coherent caching, from centrally managed approaches employing volatile storage to hybrid approaches orchestrating autonomous caches. However, due to the scale and use-case of the WLCG there is little prior work to assess the general applicability and scalability of these approaches.

Building on our previous developments of coordinated, distributed caches at KIT, we have identified the primary challenge not in the technical implementation but the underlying logic and architecture. We have studied several key issues solved by all approaches in various ways: aggregation of meta-data, identification of viable data, coherence of cache contents, and integration of temporary caches. Monitoring data from XRootD storage and HTCondor batch systems from both our Tier 1 and Tier 3 infrastructure provides realistic usage characteristics. This allows us to assess both the use cases of well-defined user jobs, as well as late-bound anonymous pilot jobs.

In this contribution, we present our findings on the implications of different architectures for distributed caching depending on the targeted use-case.

Consider for promotion No

Primary authors

Max Fischer (Karlsruhe Institute of Technology) Eileen Kuehn (Karlsruhe Institute of Technology) Rene Caspart (KIT - Karlsruhe Institute of Technology (DE)) Tabea Fessenbecker (KIT - Karlsruhe Institute of Technology (DE)) Manuel Giffels (KIT - Karlsruhe Institute of Technology (DE)) Christoph Heidecker (KIT - Karlsruhe Institute of Technology (DE))


Gunter Quast (KIT - Karlsruhe Institute of Technology (DE)) Andreas Petzold (KIT - Karlsruhe Institute of Technology (DE))

Presentation materials