Conveners
Technology talks
- Eric Vaandering (Fermi National Accelerator Lab. (US))
Technology talks
- Martin Barisits (CERN)
Technology talks
- Cedric Serfon (Brookhaven National Laboratory (US))
Technology talks
- Mario Lassnig (CERN)
Technology talks
- Eric Vaandering (Fermi National Accelerator Lab. (US))
Technology talks
- Mario Lassnig (CERN)
Disk and tape technology are continuously evolving. Rucio relies on multiple storage technologies to accommodate the data deluge from different scientific endeavors. In this talk, we'll deep dive into two storage technologies developed at CERN: EOS (disk media) and CTA (tape media), and we'll give an outlook on the current trends behind these media and what the market looks like.
This presentation will cover the latest advancements in the Rucio WebUI, offering a comprehensive overview of its current features and capabilities. In addition, we will outline the roadmap for future enhancements and planned improvements. The talk will also highlight key aspects of the interface designed to enhance the user experience. Furthermore, we will discuss how to deploy and get...
The Data Challenges are major orchestrated tests in preparation for the High-Luminosity LHC (increase by a factor of ten), with the participation of all stakeholders (multiple experiments and their data-management services, sites, networks). The second Data Challenge was conducted in February 2024. This talk will offer a summary of DC24, its goals and its achievements, with a focus on the...
In this talk, we will provide an overview of the Rucio JupyterLab extension, explaining its architecture and functionalities. We'll also discuss the latest developments and outline the roadmap for the future.
The Data Challenge 2024 was the first large-scale use of the new OAuth 2.0 token implementation in Rucio. Though declared a success, numerous concerns and open questions were voiced. This talk will offer a quick summary of the token design, then cover the lessons learned, the current efforts to refine the third-party-copy workflow, and the development on the client workflows.
Belle II experiment at KEK, Japan, has been using RUCIO as a data management service, maintained at BNL for several years. It is currently the 2nd largest RUCIO service behind ATLAS experiment at CERN in terms of the number of files stored in its catalog. RUCIO service uses the relational database underneath to store information about data that manages. For the choice of the relational...
Rucio is used to manage data by ATLAS and CMS among other scientific experiments. As Rucio does not recognise open data as a type, this is managed outside the system as it therefore likely duplicated, today this volume is 10 petabytes (85% of data in CERN open data), and it is growing. CERN experiments are mandated by CERN Open Science policy to release internal data as open data, and similar...
With the seemingly exponential growth in the volume of data in recent years, the challenges for data engineering teams in operationalizing their big data workloads (e.g. AI/ML) while ensuring access and integrity have grown increasingly more complex. More often than not, these challenges have to be surmounted with limited budgets, which can be swiftly consumed depending on the cloud storage...
A short overview of the FTS project throughout 2024, including development highlights, thoughts on the DataChallenge'24, token transition work and conclusions from the FTS & XRootd Workshop 2024.
Finally, the presentation addresses the future direction of the FTS software, exposing the main scalability and scheduling problems and how (some of) those are addressed.
The HSF Conditions Data management schema design factorises metadata management from the conditions data payloads themselves. Rucio would be a natural solution for managing replicas of those conditions data payload files. Discuss!
In this presentation, we explore the integration of cloud storage solutions within the ATLAS experiment using Rucio and FTS, with a focus on the SEAL case study. Cloud storage providers offer compelling use cases for the scientific community, such as on-demand scaling, multi-cloud Kubernetes clusters, and long-term archival options. We will discuss SEAL's current infrastructure, including...
Monitoring of Rucio has been a desired feature brought up by many communities over the years. I have developed and deployed a simplified monitoring solution that utilises Prometheus, Hermes, and Logstash to provide dashboards for communities.
A few years ago we developed a plugin that embedded the function of querying the Rucio metalinks in Xcache. This makes it much easier for users to use Xcache with Rucio managed, distributed storage. This presentation will review the existing functionalities of the plugin, and describe new works we proposed to improve the plugin. We intend to use this improved plugin at the US ATLAS analysis...
