-
Andreas Joachim Peters (CERN), Elvin Alin Sindrilaru (CERN), Luca Mascetti (CERN), Roberto Valverde Cameselle (CERN)24/04/2023, 10:00EOS Seminars
The IT storage group provides end-user access to a 700 PB disk storage system: CERN EOS.
In this seminar we will explain your possibilities to use EOS storage as a CERN user most effectively for everyone!Part 1 : Dive into the EOS eco-system
We will start with a brief introduction:
Go to contribution page
*How is the EOS service deployed and segmented? How do you get access to EOS storage and how you can... -
Andreas Joachim Peters (CERN), Elvin Alin Sindrilaru (CERN), Michael Davis (CERN)24/04/2023, 14:00
In this seminar we will go through the architecture of EOS, showcase some EOS instanace configuration and follow with an introduction to CTA!
We will explain some generic concepts, deployment models, hardware requirements, redundancy models, storage layout, scheduling & file placement - CPU, Storage and Network requirements and few tricks to optimize these. We highlight in detail the...
Go to contribution page -
Andreas Joachim Peters (CERN)25/04/2023, 09:0010 Minutes
Logistics & Information.
Go to contribution page -
Elvin Alin Sindrilaru (CERN)25/04/2023, 09:10
An overview about the developments since the last workshop.
Go to contribution page -
Cedric Caffy (CERN)25/04/2023, 09:30EOS Core Development
[To be filled]
Go to contribution page -
Abhishek Lekshmanan (CERN)25/04/2023, 09:50EOS Core Development
We take a look at the geoscheduler and see how we can introduce a new lock-free scheduling alogorithm
Go to contribution page -
Andreas Joachim Peters (CERN)25/04/2023, 10:10
This presentation will highlight the changes and improvements for the EOS filesystem access using libfuse2/3.
Go to contribution page -
Abhishek Lekshmanan (CERN)25/04/2023, 10:50
Until EOS version 5.1.8, FST metadata on FSTs were stored in a leveldb, which was often heavily contended during writes. We added a feature to move the metadata to attributes. With a minimal configuration, we should be able to switch to the new backend and FSTs automatically move from one backend to another at startup. Additionally there is some tooling to inspect all this. We briefly explain...
Go to contribution page -
Elvin Alin Sindrilaru (CERN)25/04/2023, 11:00
This talk will describe the fsck mechanism, the various options when it comes to controlling the repair process and the internal process of deciding whether a file can be fixed or not.
Go to contribution page -
Abhishek Lekshmanan (CERN)25/04/2023, 11:15
New improvements in the exisiting GroupBalancer and introduction of functionality to drain whole groups. We look at the various configuration options to run these and how these work under the hood.
Go to contribution page -
Elvin Alin Sindrilaru (CERN)25/04/2023, 11:25EOS Core Development
This presentation gives an overview of the token support in EOS. We'll discuss the configuration options, what plugins need to be enabled for the various protocols and how to configure them. Besides this, we'll trace one particular request using tokens to see how it interacts with the existing authentication/authorization features that already exists in EOS and provide some helpful examples.
Go to contribution page -
Andreas Joachim Peters (CERN)25/04/2023, 11:40
This presentation includes performance benchmarks comparing local and remote IO for various use-cases, storage stacks and protocols.
Go to contribution page -
LI Haibo lihaibo25/04/2023, 13:45
The Institute of High Energy Physics undertakes many large scientific engineering projects in China. These large scientific projects generate a large amount of data every year and require a computing platform for analysis and processing.
Go to contribution page
EOS is one of the main storage system at IHEP since 2016. The EOS instance at IHEP has currently a gross capacity of 50 PB. Currently we have deployed 6... -
Armin Burger (JRC)25/04/2023, 14:00
The Joint Research Centre (JRC) of the European Commission is running the Big Data Analytics Platform (BDAP) to enable the JRC projects to store, process, and analyze a wide range of data. The platform evolved as a core service for JRC scientists to produce knowledge and insights in support of EU policy making.
EOS is the main storage system of the BDAP for scientific data. It is in...
Go to contribution page -
Xuantong Zhang (Chinese Academy of Sciences (CN))25/04/2023, 14:20
The EOS system serving as a grid storage element at IHEP, CAS started since 2021, working for JUNO experiment. A CTA with its EOS SE buffer also started its service for JUNO since 2023. In this talk, we would like to share our experiences and thoughts about the SE operations, including deployment, monitoring, data transfer performance, authentication management with VOMS and Sci-token,...
Go to contribution page -
Dr Emmanouil Vamvakopoulos (Université Paris-Saclay (FR))25/04/2023, 14:40
to.be.specified
Go to contribution page -
Dr Yaodong Cheng (Institute of High Energy Physics, Chinese Academy of Sciences)25/04/2023, 14:55EOS Core Development
Computational storage involves integrating compute resources with storage devices or systems to enable data processing within the storage device. This approach reduces data movement, enhances processing efficiency, and reduces costs. To facilitate in-situ data processing on storage servers, we developed a computational storage plugin that can be added to EOS FST. This plugin enables users to...
Go to contribution page -
Stefan Piperov (Purdue University (US))25/04/2023, 15:15
In 2022 the CMS Tier-2 at Purdue University migrated its 10PB storage system from HDFS to EOS. Here we report on the details of the process, the difficulties we encountered and the ways in which we solved them. We also report on the current status of the storage system, and our future plans.
Go to contribution page -
Dan Szkola (Fermi National Accelerator Lab. (US))25/04/2023, 16:00
Fermilab has been running an EOS instance since testing began in June 2012. By May 2013, before becoming production storage, there was 600TB allocated for EOS. Today, there is approximately 13PB of storage available in the EOS instance.
The LPC cluster is a 4500-core user analysis cluster with 13 PB of EOS storage. The LPC cluster supports several hundred active CMS users at any given...
Go to contribution page -
Ryan Taylor (University of Victoria (CA))25/04/2023, 16:10
I will discuss our efforts to deploy EOS on Kubernetes at the University of Victoria T2 site for ATLAS, using a Helm chart and CephFS storage.
Go to contribution page -
Sang Un Ahn (Korea Institute of Science & Technology Information (KR))25/04/2023, 16:20Sites and Deployments
We present the current operation status of CDS (the Disk-based Custodial Storage) for ALICE experiment. The CDS is based on EOS Erasure Coding implementation with four parity mode to match with Tape based archival storage in terms of data protection. We will discuss briefly the plan of CDS operation automation for hardware intervention, especially the disk replacement, and of its expansion to...
Go to contribution page -
Cristian Contescu (CERN)25/04/2023, 16:40
2022 was a critical year which had some operational impact on all systems and services. In this talk we are going to present a few operational decisions, their implication on the EOS storage for the ALICE data taking and how we implemented mitigations by bringing improvements to the operations model as well as to the software stack.
Go to contribution page -
Michael Davis (CERN)26/04/2023, 09:00
Welcome to the second annual CTA Day at the EOS Workshop!
In mid-2022, the LHC awoke from its second Long Shutdown and restarted physics operations. Run-3 data-taking rates are several times higher than during Run-2; already CTA hit a new record of 26 PB archived in one month in November 2022.
This presentation will introduce the CTA Project, Team and Community, as well as an overview of...
Go to contribution page -
Mark Harvey26/04/2023, 09:10
Virtual Tape Libraries (VTLs) have been widely used in data storage environments. mhVTL design goals are unique in it is not attempting to be a better tape library, simply emulate real hardware—for those situations where it is impractical to carry a physical tape library with you.
Go to contribution page -
Denis Sergeevich Lujanski26/04/2023, 09:25
Since 2021, AARNet have been using restic backup software in conjunction with CTA to backup user data in production EOS clusters. The road to production has not been without its challenges, requiring us to modify restic and create a custom backup scheduler and client workflows. This presentation will aim to cover the architecture, with a focus on restic: the basics, the customisations and...
Go to contribution page -
George Patargias26/04/2023, 09:40
Antares is the new tape archive service at RAL Tier-1 that went into production on 4th of March 2022. The service is built around the EOS/CTA technologies developed at CERN. EOS is the user facing service that manages the incoming namespace requests and a thin SSD buffer, and CTA is deployed as the tape back-end system. In this talk, we describe the setup of ANTARES and discuss the service’s...
Go to contribution page -
Dr Yujiang BI (Institute of High Energy Physics, Chinese Academy of Sciences)26/04/2023, 09:55
We will share our experiences on EOS CTA and talk about our plan for the future of CTA. All experiments of IHEP have adopted CTA as the main tape storage management system, and preparing a new tape library for TIER1 of LHCb. We've test the tape restful API with X509 and token auth to access EOS & CTA via HTTP as well as XRootD. In the future, we shall upgrade our production instances to EOS &...
Go to contribution page -
Eric Vaandering (Fermi National Accelerator Lab. (US))26/04/2023, 10:15
Fermilab has decided to replace Enstore, its locally developed tape management system, with CTA. Fermilab runs two Enstore instances: CMS with a small, dedicated tape buffer and Public operating like an HSM with tight integration between dCache and Enstore
This talk will cover:
- Metadata migration from Enstore to CTA
- Results of dCache integration with CTA at FNAL
- Performance...
-
Mwai Karimi (DESY)26/04/2023, 10:50
Since early 2021 CTA has been on a test bed at DESY. Having observed no flies in the ointment and seamless integration with dCache in-place, CTA advances to production in 2023. This presentation will give an overview of the current migration and deployment status as well as future plans at DESY.
Go to contribution page -
Mrs Elisabet Carrasco Santos (PIC-IFAE)26/04/2023, 11:05
At PIC, we currently have Enstore as our tape storage system, but due to the discontinuation of its support and development in the near future, we want to share our experiences and insights about the testing and implementation of Cern Tape Archive (CTA) as a potential replacement.
We have set up a CTA test instance integrated with dCache in order to evaluate its functionalities and work on...
Go to contribution page -
Thomas Byrne26/04/2023, 11:20
At RAL, we intend to consolidate the two CASTOR instances into a single CTA instance with multiple EOS disk buffers, similar to the CERN architecture. The 'WLCGTape' Castor instance has been fully migrated to CTA at RAL, and has been running in production for LHC run 3 data taking.
In preparation for the migration of the 'Facilities' CASTOR instance at RAL onto our CTA instance, there have...
Go to contribution page -
Julien Leduc (CERN)26/04/2023, 11:35
An EOSCTA instance is an EOS instance - commonly called a tape buffer - configured with a CERN Tape Archive (CTA) back-end.
Go to contribution page
This EOS instance is entirely bandwidth oriented: it offers an SSD based tape interconnection, it can contain spinning disks if needed and it is optimized for the various tape workflows.
This talk will present how to enable EOS for tape using CTA and the Swiss horology... -
Volodymyr Yurchenko (CERN)26/04/2023, 11:50
The CERN Tape Archive (CTA) is a vital system for storing and retrieving data at CERN. However, the reliability of the CTA system can be impacted by various factors, including hardware failures, software bugs, and network connectivity issues. To ensure the continued availability of the stored data, it is critical to have robust mechanisms in place for handling failed requests.
This talk...
Go to contribution page -
Richard Bachmann (CERN)26/04/2023, 12:05
Reliable and effective monitoring is essential for smooth operations and for tailoring an EOSCTA deployment to users’ needs. Short-term monitoring provides alerting for abnormal system states, and long-term monitoring allows us to track system usage and performance over time.
In this presentation we walk you through the general setup we use for Tier-0 storage at CERN, which allows us to...
Go to contribution page -
Vladimir Bahyl (CERN)26/04/2023, 12:20
A ‘Repack’ is the process of moving or copying data from one tape cartridge to one or multiple others. Such a process may be needed for various reasons, such as transferring data to more compact media, creating additional copies, and recovering data from faulty media. At CERN the CTA software manages the data transfer itself, but more steps are needed in order to complete the full repack...
Go to contribution page -
Julien Leduc (CERN)26/04/2023, 14:00CTA
This hands-on session will focus on installing and configuring a standalone CTA CI runner:
- single host kubernetes cluster in Alma9
- 1 Virtual tape library with CTA CI requirements
At the end of this sessions the participants should be able to run CTA Continuous Integration test on their box.
Go to contribution page -
Lasse Tjernaes Wardenaer (Norwegian University of Science and Technology (NTNU) (NO))26/04/2023, 15:50
When updating the disk file meta data for tape files, it is necessary to do the updates in both EOS and CTA. Examples of use-cases that require these updates are migration to CTA, moving a file from one EOS instance to another, switching from single to dual copy and restoring deleted files.
The tools for handling these use-cases are not atomic, but they are idempotent and consistency is...
Go to contribution page -
Joao Afonso (CERN)26/04/2023, 16:05
As the amount of stored data, scope and operation complexity of CTA grew, it became necessary to improve the level of control over each tape lifecycle and to provide mechanisms that allow for an improved automation of CTA operations, such as repacking.
In this presentation we will talk about the new CTA tape states, which expand the behaviour of the disabled state. This includes support for...
Go to contribution page -
Jorge Camarero Vera (CERN)26/04/2023, 16:20
At the EOS Workshop 2022 BoF, it was decided that CTA should add support for reading OSM/dCache and Enstore tape formats. To make this feature work seamlessly within CTA, we refactored our codebase to accommodate different tape file readers.
In this presentation, we will discuss the design and implementation of external tape format readers into CTA. We will also cover the unit and...
Go to contribution page -
Michael Davis (CERN)26/04/2023, 16:35
CTA software development is primarily driven by the needs of the CERN experimental programme. Looking beyond Run-3, data rates are set to continue to rise exponentially into Run-4 and beyond. The CTA team are planning how to scale the software and service to meet these new challenges.
CTA is also driven by the needs of the community outside CERN. The landscape of tape archival for...
Go to contribution page -
Michael Davis (CERN)26/04/2023, 16:50
Final comments, questions and discussion. Segue into the apéro in R2 where we can continue talking.
Go to contribution page -
Guilherme Amadio (CERN)27/04/2023, 09:00
Latest updates about XRootD development and the March XRootD Workshop.
Go to contribution page -
Manuel Reis (Universidade de Lisboa (PT))27/04/2023, 09:15
A report about the deployment of the major version update of the eos client stack. From which XRootD v5 and Fuse v3 upgrades standout.
Go to contribution page -
Roberto Valverde Cameselle (CERN)27/04/2023, 09:30
Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. CERN EOS operations team have developed a Prometheus exporter for EOS, that exposes common EOS metrics in prometheus format. This presentation...
Go to contribution page -
Cedric Caffy (CERN)27/04/2023, 09:40
[To be filled]
Go to contribution page -
Dr Maria Arsuaga Rios (CERN)27/04/2023, 09:5520 Minutes
-
Gregor Molan (Comtrade 360's AI Lab)27/04/2023, 10:15
The Graphical User Interface (GUI) for CERN EOS could be crucial in the interaction between potential users and the EOS storage technology. The GUI could serve as an interface between a user and the complex EOS infrastructure, enabling non-experts to learn and discover EOS features seamlessly and effectively. This would help users interact with the storage infrastructure without needing to...
Go to contribution page -
Branko Blagojevic (Comtrade)27/04/2023, 10:50
Context
An overview of the EOS Windows native client for EOS users on Windows operating systems
Objectives
EOS Windows native client should provide Windows platform users with native access to EOS cluster for both file transferring and command requests, giving them improved user experience compared to EOS Linux client.
Method
EOS Windows native client comes with two...
Go to contribution page -
Ivan Arizanovic (Comtrade 360's AI Lab)27/04/2023, 11:00
EOS-drive is part of the EOS Windows Native Client package, it mounts the EOS filesystem as a Windows disk drive by which Windows applications interact with the EOS filesystem.
EOS-drive communicates with Windows applications through the user-mode Dokan library and kernel-mode Dokan driver. File operation requests from applications (e.g., CreateFile, ReadFile, WriteFile...) are sent to the...
Go to contribution page -
Diogo Castro (CERN)27/04/2023, 11:10
CERNBox combines the ease of use of a file sync and share service with the power of the scientific data processing infrastructure at CERN. Built on top of EOS and ownCloud, it provides a simple and uniform way to access over 15PB of research, administrative and engineering data across more than 2 billion files.
In this talk we will go through the latest advances made possible with the new...
Go to contribution page -
Gianmaria Del Monte (CERN)27/04/2023, 11:3015 Minutes
The IT storage group at CERN is resposible to ensure integrity and security of all the stored data for physics and general computing services. In the last years a backup orchestrator, cback, has been developed based on the open source backup software restic. Cback is able to backup EOS, CephFS and any local mountable file system, like NFS or DFS. cback is currently used to daily backup CERNBox...
Go to contribution page -
Emmanouil Bagakis (CERN), Roberto Valverde Cameselle (CERN)27/04/2023, 11:45
EOS for users service, internally known as EOSHPM (EOSHOME, EOSPROJECT and EOSMEDIA) currently stores 2.8 billion files and more than 20PB of storage. We store data of more than 45,000 users and project spaces and host multimedia related use cases for the IT Department. Data is accessed via filesystem (fuse), CERNBox (Web interface, Sync/Mobile client), SAMBA and HTTP. We will be reporting on...
Go to contribution page -
Andreas Joachim Peters (CERN), Elvin Alin Sindrilaru (CERN)27/04/2023, 11:55
We will give an overview about the development roadmap for 2023 and beyond.
Go to contribution page -
Hugo Gonzalez Labrador (CERN)27/04/2023, 14:00
Choose timezone
Your profile timezone: