Speaker
Description
The interTwin project, funded by Horizon Europe, developed a Digital Twin Engine (DTE), a platform for the development and running of Digital Twins across multiple scientific domains. A central component of the DTE is the interTwin Data Lake, a federated storage layer that integrates HPC, HTC, and cloud-based datasets and provides unified access while preserving site-local policies and permissions. The interTwin Data Lake is based on Rucio and FTS.
The project identified access to existing storage as a barrier to data lake adoption. To tackle this, the project developed two new components: Teapot and ALISE. Together these enable scalable and secure access to the Data Lake in HPC and HTC environments. They achieve this by providing automated, bulk access to site storage while mapping each request to a site-local account, without requiring centrally managed accounts.
Teapot is a multi-tenant WebDAV service built on StoRM-WebDAV and provides integration of HPC and HTC storage into the federated Data Lake. Its architecture preserves file ownership and enforces native filesystem permissions, allowing sites to expose storage resources without altering local policies. Teapot has enabled CESGA and KBFI to join the Data Lake, with additional sites in progress. By enabling HPC/HTC storage integration into the Data Lake, it supports Digital Twin workflows that require bulk data staging and processing on HPC and HTC resources.
Teapot integrates with ALISE, a lightweight user enrolement service that supports linking external- and site-local identities. ALISE makes this mapping information available to services, enabling the site to support OIDC-based authentication while retaining local account and authorization models.