Parallel file systems have reach new height in performance and scale. Storage systems delivering in the 1+TB/sec at a 100+PB scale are now available in several HPC environments including enterprise ones.
It went with efforts, sweat if not tears, but with little surprise since the performance community has such a long track record of success in challenging the new order of magnitude....
Onedata is a global high-performance data management system that unifies data access across globally distributed environments and multiple types of underlying storages, such as NFS, Lustre, GPFS, Amazon S3, CEPH, as well as other POSIX-compliant file systems. It allows users to share, collaborate and perform computations on their data. Due to its fully distributed architecture, Onedata enables...
Every year at CS3 we all come together to talk about the things we've built and how they've grown - more users, more files, more shares, more storage used than in past years, more features we've added. Last year, we introduced a particularly interesting feature to the AARNet CloudStor ecosystem: S3 gateways as a means of convenient, high-speed data transfer directly to our backend storage....
SpectrumScale is a software defined parallel file system, which can scale over multiple nodes, networks and block storage types. SpectrumScale R5.x supports Watchfolder, which is somehow comparable to linux inotify, but WF supports to be used over multiple directories and sub-trees of a file system and even over the complete name space recursively.
Based on Watchfolder, NEXTCLOUD and IBM...
As commercial, governmental, and research organizations continue to move from manual pipelines to automated processing of their vast and growing datasets, they are struggling to find meaning in their repositories.
Many products and approaches now provide data discoverability through indexing and aggregate counts, but few also provide the level of confidence needed for making strong...
Building a scalable public cloud platform for hundred of thousands from scratch can be a difficult and challenging task.
This session will cover a short introduction of luckycloud and how we integrated and fully automated the deployment of Seafile clusters with highly scalable multi petabyte storage backends. We will also show how we build a reliable and powerful storage backend for our...
We are a Nordic cloud provider that have been operating Storage as a Service running on ceph clusters for the last three years - providing storage service to the academic sector in Sweden and Norway. In this session I will share some of our experience. I will not go deeply into technical details, but I will rather share some lessons we have learnt about how to build a good team, how we...
Consider a 100 TB NFS data-set on your on-premises file server that you need to import in Azure Blob storage for further processing using Azure Machine Learning Studio and you need the data there fast.
Also consider having this repeated several times with slightly changed data-sets.
Some might consider this to be a challenge.
With Cloud Sync NetApp offers:
• a fully managed easy-to-use,...
As technologies continue to evolve, the size and amount of data that your organization must work with is growing exponentially.
Keeping ahead of this data growth requires a scalable and innovative high-performance solution with a lightning-fast, highly reliable IT infrastructure to process, store, and analyse your data. However, the cost and complexity of deploying and operating an HPC...
Commercial services for Digital Preservation that are currently available have not been proven to scale to the "petabyte region and beyond", not address the complex data types, often domain-specific, that are needed by many scientific disciplines. In-house services, where they exist have often not acquired the degree of “trustworthiness” verified through certification schemes.
Using a...
Sync&share systems are widely used at universities and commercial institutions in order to address data storage and sharing as well as data synchronisation needs. Academic users mostly use open source solutions, while companies, especially SMEs prefer commercial products with paid support.
PSNC decided to use Seafile, a scalable, purpose-made, reliable and performant sync&share system. The...
