Conveners
Scalable Storage Backends
- Hugo Gonzalez Labrador (CERN)
Research Drive, the Dutch Sync & Share service based on ownCloud, uses OpenStack Swift S3 as its storage backend. Where the integration of S3 within the software is not that good, we will migrate back to a posix compliant file system, namely CephFS. But how to migrate almost 2 PB of data without too much downtime...
From 1VM+LUN to k8s+S3 - an uneasy way…
Since 2015 PSNC has provided a sync & share service for science and academia in Poland, based on the Seafile software. We started small by running a 1VM+1LUN setup and community version of the software, integrated with the local PSNC’s LDAP. In 2016 we began to build a fully-fledged setup based on a cluster of application servers, background jobs...
Different high-performance, high-available file systems can store big data (hundreds of PB) and provide high data throughput (hundreds of TB per second). Each of these solutions highlights its advantages, and it is challenging to compare them.
Based on 30 years of storage development experience, Comtrade provided test scenarios to compare these file systems. On the appropriate...
The IT storage group at CERN is resposible to ensure integrity and security of all the stored data for physics and general computing services. In the last years a backup orchestrator, cback, has been developed based on the open source backup software restic. Cback is able to backup EOS, CephFS and any local mountable file system, like NFS or DFS. cback is currently used to daily backup CERNBox...
Data are said to live forever, however their life is a complex journey. Initiated at acquisition or production date, data start a whole life cycle. During the different epochs of this life cycle, data will be moved, processed, compressed, shipped, archived.
To ease the management of this data orchestration, modern storage systems provide powerful tools. The foundation of these tools remains...
The open source iRODS (Integrated Rule-Oriented Data System) data management platform presents a virtual filesystem, metadata catalog, and policy engine designed to give organizations maximum control and flexibility over their data management practices and enforcement. Since iRODS defines its own RPC API and protocol, interoperability with other software has always lagged new server features...
