The IT storage group provides end-user access to a 700 PB disk storage system: CERN EOS.
In this seminar we will explain your possibilities to use EOS storage as a CERN user most effectively for everyone!
Part 1 : Dive into the EOS eco-system
We will start with a brief introduction:
*How is the EOS service deployed and segmented? How do you get access to EOS storage and how you can...
In this seminar we will go through the architecture of EOS, showcase some EOS instanace configuration and follow with an introduction to CTA!
We will explain some generic concepts, deployment models, hardware requirements, redundancy models, storage layout, scheduling & file placement - CPU, Storage and Network requirements and few tricks to optimize these. We highlight in detail the...
An overview about the developments since the last workshop.
We take a look at the geoscheduler and see how we can introduce a new lock-free scheduling alogorithm
This presentation will highlight the changes and improvements for the EOS filesystem access using libfuse2/3.
Until EOS version 5.1.8, FST metadata on FSTs were stored in a leveldb, which was often heavily contended during writes. We added a feature to move the metadata to attributes. With a minimal configuration, we should be able to switch to the new backend and FSTs automatically move from one backend to another at startup. Additionally there is some tooling to inspect all this. We briefly explain...
This talk will describe the fsck mechanism, the various options when it comes to controlling the repair process and the internal process of deciding whether a file can be fixed or not.
New improvements in the exisiting GroupBalancer and introduction of functionality to drain whole groups. We look at the various configuration options to run these and how these work under the hood.
This presentation gives an overview of the token support in EOS. We'll discuss the configuration options, what plugins need to be enabled for the various protocols and how to configure them. Besides this, we'll trace one particular request using tokens to see how it interacts with the existing authentication/authorization features that already exists in EOS and provide some helpful examples.
This presentation includes performance benchmarks comparing local and remote IO for various use-cases, storage stacks and protocols.
The EOS system serving as a grid storage element at IHEP, CAS started since 2021, working for JUNO experiment. A CTA with its EOS SE buffer also started its service for JUNO since 2023. In this talk, we would like to share our experiences and thoughts about the SE operations, including deployment, monitoring, data transfer performance, authentication management with VOMS and Sci-token,...
Computational storage involves integrating compute resources with storage devices or systems to enable data processing within the storage device. This approach reduces data movement, enhances processing efficiency, and reduces costs. To facilitate in-situ data processing on storage servers, we developed a computational storage plugin that can be added to EOS FST. This plugin enables users to...
In 2022 the CMS Tier-2 at Purdue University migrated its 10PB storage system from HDFS to EOS. Here we report on the details of the process, the difficulties we encountered and the ways in which we solved them. We also report on the current status of the storage system, and our future plans.
Fermilab has been running an EOS instance since testing began in June 2012. By May 2013, before becoming production storage, there was 600TB allocated for EOS. Today, there is approximately 13PB of storage available in the EOS instance.
The LPC cluster is a 4500-core user analysis cluster with 13 PB of EOS storage. The LPC cluster supports several hundred active CMS users at any given...
I will discuss our efforts to deploy EOS on Kubernetes at the University of Victoria T2 site for ATLAS, using a Helm chart and CephFS storage.
We present the current operation status of CDS (the Disk-based Custodial Storage) for ALICE experiment. The CDS is based on EOS Erasure Coding implementation with four parity mode to match with Tape based archival storage in terms of data protection. We will discuss briefly the plan of CDS operation automation for hardware intervention, especially the disk replacement, and of its expansion to...
2022 was a critical year which had some operational impact on all systems and services. In this talk we are going to present a few operational decisions, their implication on the EOS storage for the ALICE data taking and how we implemented mitigations by bringing improvements to the operations model as well as to the software stack.
Since 2021, AARNet have been using restic backup software in conjunction with CTA to backup user data in production EOS clusters. The road to production has not been without its challenges, requiring us to modify restic and create a custom backup scheduler and client workflows. This presentation will aim to cover the architecture, with a focus on restic: the basics, the customisations and...
We will share our experiences on EOS CTA and talk about our plan for the future of CTA. All experiments of IHEP have adopted CTA as the main tape storage management system, and preparing a new tape library for TIER1 of LHCb. We've test the tape restful API with X509 and token auth to access EOS & CTA via HTTP as well as XRootD. In the future, we shall upgrade our production instances to EOS &...
Fermilab has decided to replace Enstore, its locally developed tape management system, with CTA. Fermilab runs two Enstore instances: CMS with a small, dedicated tape buffer and Public operating like an HSM with tight integration between dCache and Enstore
This talk will cover:
- Metadata migration from Enstore to CTA
- Results of dCache integration with CTA at FNAL
- Performance...
Since early 2021 CTA has been on a test bed at DESY. Having observed no flies in the ointment and seamless integration with dCache in-place, CTA advances to production in 2023. This presentation will give an overview of the current migration and deployment status as well as future plans at DESY.
At PIC, we currently have Enstore as our tape storage system, but due to the discontinuation of its support and development in the near future, we want to share our experiences and insights about the testing and implementation of Cern Tape Archive (CTA) as a potential replacement.
We have set up a CTA test instance integrated with dCache in order to evaluate its functionalities and work on...
The CERN Tape Archive (CTA) is a vital system for storing and retrieving data at CERN. However, the reliability of the CTA system can be impacted by various factors, including hardware failures, software bugs, and network connectivity issues. To ensure the continued availability of the stored data, it is critical to have robust mechanisms in place for handling failed requests.
This talk...
This hands-on session will focus on installing and configuring a standalone CTA CI runner:
- single host kubernetes cluster in Alma9
- 1 Virtual tape library with CTA CI requirements
At the end of this sessions the participants should be able to run CTA Continuous Integration test on their box.
When updating the disk file meta data for tape files, it is necessary to do the updates in both EOS and CTA. Examples of use-cases that require these updates are migration to CTA, moving a file from one EOS instance to another, switching from single to dual copy and restoring deleted files.
The tools for handling these use-cases are not atomic, but they are idempotent and consistency is...
At the EOS Workshop 2022 BoF, it was decided that CTA should add support for reading OSM/dCache and Enstore tape formats. To make this feature work seamlessly within CTA, we refactored our codebase to accommodate different tape file readers.
In this presentation, we will discuss the design and implementation of external tape format readers into CTA. We will also cover the unit and...
Final comments, questions and discussion. Segue into the apéro in R2 where we can continue talking.
Latest updates about XRootD development and the March XRootD Workshop.
A report about the deployment of the major version update of the eos client stack. From which XRootD v5 and Fuse v3 upgrades standout.
Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. CERN EOS operations team have developed a Prometheus exporter for EOS, that exposes common EOS metrics in prometheus format. This presentation...
The Graphical User Interface (GUI) for CERN EOS could be crucial in the interaction between potential users and the EOS storage technology. The GUI could serve as an interface between a user and the complex EOS infrastructure, enabling non-experts to learn and discover EOS features seamlessly and effectively. This would help users interact with the storage infrastructure without needing to...
Context
An overview of the EOS Windows native client for EOS users on Windows operating systems
Objectives
EOS Windows native client should provide Windows platform users with native access to EOS cluster for both file transferring and command requests, giving them improved user experience compared to EOS Linux client.
Method
EOS Windows native client comes with two...
EOS-drive is part of the EOS Windows Native Client package, it mounts the EOS filesystem as a Windows disk drive by which Windows applications interact with the EOS filesystem.
EOS-drive communicates with Windows applications through the user-mode Dokan library and kernel-mode Dokan driver. File operation requests from applications (e.g., CreateFile, ReadFile, WriteFile...) are sent to the...
The IT storage group at CERN is resposible to ensure integrity and security of all the stored data for physics and general computing services. In the last years a backup orchestrator, cback, has been developed based on the open source backup software restic. Cback is able to backup EOS, CephFS and any local mountable file system, like NFS or DFS. cback is currently used to daily backup CERNBox...
EOS for users service, internally known as EOSHPM (EOSHOME, EOSPROJECT and EOSMEDIA) currently stores 2.8 billion files and more than 20PB of storage. We store data of more than 45,000 users and project spaces and host multimedia related use cases for the IT Department. Data is accessed via filesystem (fuse), CERNBox (Web interface, Sync/Mobile client), SAMBA and HTTP. We will be reporting on...
We will give an overview about the development roadmap for 2023 and beyond.
