4th Rucio Community Workshop (Virtual)

28 Sept 2021, 15:00 → 30 Sept 2021, 18:00 Europe/Zurich

Rucio is a software framework that provides functionality to organize, manage, and access large volumes of scientific data using customisable policies. The data can be spread across globally distributed locations and across heterogeneous data centers, uniting different storage and network technologies as a single federated entity. Rucio offers advanced features such as distributed data recovery or adaptive replication, and is highly scalable, modular, and extensible. Rucio has been originally developed to meet the requirements of the high-energy physics experiment ATLAS, and is continuously extended to support LHC experiments and other diverse scientific communities.

We set up a mailing list to which you can subscribe and where we will send more details about the program in the coming weeks.

We also created a Slack channel dedicated to the workshop discussion on the Rucio Slack workspace (Invitation Link). Join #workshop.

Group Photo.png

Tuesday, 28 September

15:00 → 15:45 Opening & Closing

15:00 Welcome

Speaker: Martin Barisits (CERN)

Recording

Welcome 2021.pdf

15:05 Agenda & Logistics

Speaker: Mario Lassnig (CERN)

Agenda & Logistics.pdf

Recording

15:15 Rucio: State of the Union

Speaker: Martin Barisits (CERN)

Recording

Rucio State of the Union.pdf

15:45 → 16:30 Community Reports

Convener: Alastair Dewhurst (Science and Technology Facilities Council STFC (GB))

15:45 CMS

Speaker: Eric Vaandering (Fermi National Accelerator Lab. (US))

CMS Rucio WS 2021.pdf

Recording

16:00 Rucio for the Light Dark Matter eXperiment (LDMX)

The Light Dark Matter eXperiment (LDMX) is a planned small-scale accelerator-based experiment to search for dark matter in the sub-GeV mass region. Finalizing the design of the detector relies on Monte-Carlo simulation of expected physics processes. A distributed computing pilot project was initiated based around existing software used by other communities, including Rucio for data management. Rucio is primarily used as a dataset catalog, and LDMX also makes extensive use of Rucio metadata to record physics properties as well as bookkeeping information on the data produced. In this talk we describe how Rucio is used by LDMX and propose some possible extensions to the metadata functionality.

Speaker: David Cameron (University of Oslo (NO))

Recording

Rucio for LDMX.pdf

16:15 DUNE Rucio Usage and Status

We present the current state of the DUNE Rucio deployment and plans for extension and expansion. Results will include new monitoring and testing that has been
added as well as plans for moving from the current hybrid system to a native
Rucio deployment.

Speaker: Steven Timm (Fermi National Accelerator Lab. (US))

Recording

rucio_wkshp_2021sep28.pdf

rucio_wkshp_2021sep28.pptx

16:30 → 17:00 Break

17:00 → 18:00 Community Reports

Convener: Mario Lassnig (CERN)

17:00 Customizing Rucio at LCLS

This talk will present the speaker's onboarding experience as someone new to Rucio, covering the aspects of using documentation, building and standing up the containers, and configuring the Rucio system. Furthermore, specifics will be discussed in regard to the customization in development to meet the project requirements for LCLS, a free electron laser that produces ultra fast X-ray pulses.

Speaker: Kenny Lo (SLAC National Accelerator Laboratory)

LCLS_4thRucioWorkshop.pdf

Recording

17:15 IGWN

Speaker: Gabriele Gaetano Fronze' (INFN Torino (IT) and LIGO-Virgo-Kagra Collaboration (US/IT/JP))

Recording

RucioWorkshop2021-min.pdf

17:30 FTS: Updates, Direction and Plans

Speaker: Mihai Patrascoiu (CERN)

FTS_Rucio_Workshop_2021.pdf

Recording

17:45 AAI/Tokens IAM

Speaker: Andrea Ceccanti (Universita e INFN, Bologna (IT))

20210928-IAM-Rucio-Workshop.pdf

Recording

18:00 → 19:00 Panels: Rucio in a non-grid environment

Convener: Cedric Serfon (Brookhaven National Laboratory (US))

18:00 Rucio in a non-grid environment panel

Speakers: Alastair Dewhurst (Science and Technology Facilities Council STFC (GB)), Andrea Manzi, David Cameron (University of Oslo (NO)), Ilija Vukotic (University of Chicago (US)), Mario Lassnig (CERN), Oliver Keeble (CERN)

Wednesday, 29 September

09:15 → 10:00 Keynote

Convener: Mario Lassnig (CERN)

09:15 Pipe Dreams (and Nightmares)

Australia's Academic and Research Network (AARNet) was established in 1989 and is widely regarded as the founder of the Internet in Australia and renowned as the architect, builder and operator of world-class network infrastructure for research and education.
We are Australia's National Research and Education Network (NREN). We connect over one million users—researchers, faculty, staff and students—at institutions across Australia, supporting education and research across a diverse range of disciplines including high energy physics, climate science, genomics, radio astronomy and the arts.
Nationally, AARNet interconnects Australian universities, the CSIRO, and other organisations who have a research and education mission, or with whom the education and research sector interacts. These include hospitals, vocational training providers, schools and museums. Internationally, AARNet interconnects the Australian Research and Education (R & E) community to the world – and continuously develops new capabilities and partnerships to facilitate seamless data access and transfer.
Today, we'll talk about some of the work we've done in the data access and transfer space, challenges we've faced, and our plans for the future.

Speaker: Crystal Michelle Chua

cc_ga_rucio4.pdf

Recording

10:00 → 11:05 Community Reports: WFMS

Convener: Cedric Serfon (Brookhaven National Laboratory (US))

10:00 Panda

Speaker: Paul Nilsson (Brookhaven National Laboratory (US))

PanDA and Rucio, September 29, 2021.pdf

Recording

10:15 DIRAC

Speaker: Federico Stagni (CERN)

Recording

RucioWS.pdf

10:30 CMS Workload Management and Rucio integration

Speaker: Todor Trendafilov Ivanov (University of Sofia - St. Kliment Ohridski (BG))

Recording

Rucio at WMCore.pdf

10:45 Discussion

11:05 → 11:25 Break

11:25 → 12:25 Community Reports

Convener: Martin Barisits (CERN)

11:25 Belle II

Speaker: Cedric Serfon (Brookhaven National Laboratory (US))

2021-09-29 - Rucio and Belle II.pdf

Recording

11:40 Rucio and ScienceMesh: Enabling data management for the CS3 community

Speakers: Giuseppe Lo Presti (CERN), Rahul Chauhan

Chauhan, LoPresti - ScienceMesh - Rucio workshop 2021.pdf

Recording

11:55 ATLAS

Speaker: Mario Lassnig (CERN)

Recording

Rucio @ ATLAS.pdf

15:00 → 16:40 Community Reports: Astronomy

Convener: Rosie Bolton (SKA Organisation)

15:00 Session Introduction

Speaker: Rosie Bolton (SKA Organisation)

Recording

15:05 SKA Rucio deployment and metadata/ data lifecycle use case

Speaker: Rohini Joshi (SKA Organisation)

Recording

SKAO - Rucio Community Workshop.pdf

15:20 MAGIC telescope rucio deployment

Speaker: Agustin Bruzzese

ESCAPE-2021-09-29.pdf

Recording

15:35 CTA rucio use case and development with cloud storage

Speaker: Frederic Gillardo (Centre National de la Recherche Scientifique (FR))

Recording

RucioUseCase-2021.pdf

RucioUseCase-2021.pptx

15:50 LOFAR Use Cases in ESCAPE - Experience and Future Directions

We will be presenting our experiences and results for LOFAR use case using ESCAPE data lake and Rucio. Reflections on future requirements including for other data sets (e.g. APERTIF) shall also be deliberated.

Speaker: Yan Grange (ASTRON)

Recording

Rucio for LOFAR.pdf

16:05 IVOA

Speaker: Dave Morris

20210929-ESCAPE-Rucio.pdf

Recording

16:20 Rucio @ SDC/KIS

Speaker: Peter Caligari (KIS)

2021_09_4thRucioWS21_PC.pdf

Recording

16:40 → 17:00 Break

17:00 → 18:00 Community Reports

Convener: Eric Vaandering (Fermi National Accelerator Lab. (US))

17:00 ESCAPE Data Lake as a Service

Experiments and scientists, whether in the process of designing and building up a data management system or managing multi-petabyte data historically, gather in the European Science Cluster of Astronomy & Particle physics ESFRI research infrastructures (ESCAPE) project to address computing challenges by developing common solutions in the context of the EOSC.
A modular ecosystem of services and tools constitutes the ESCAPE Data Lake, which is exploited by flagship ESFRIs in Astro-particle Physics, Electromagnetic and Gravitational-Wave Astronomy, Particle Physics, and Nuclear Physics to pursue together the FAIR and open-access data principles.
This infrastructure fulfils the needs of the ESCAPE community in terms of data organisation, management, and access, and dedicated assessment exercises demonstrated its robustness.
As a result, collaborating sciences are choosing their reference implementations of the various technologies among the proposed solutions.
A variety of challenges and specific use cases boost ESCAPE to carefully take into account both user and infrastructure perspectives, and contributed to successfully conclude the pilot phase beyond expectations, embarking on a like-production prototype stage.
The ongoing phase of the project aims at consolidating the functionalities of the services, e.g. integrating token-based AuthN/Z or deploying a tailored content delivery and caching layer, and at simplifying the user experience. Specifically for this reason, a considerable effort is being devoted towards a DataLake-as-a-Service whose goal is to provide the end-user with a Notebook ready-to-be-used and fully integrated with the Data Lake.
ESCAPE milestones achieved during the length of the project represent a fundamental accomplishment under both sociological and computing model aspects for different scientific communities that should address upcoming data management and computing challenges in the next decade.

Speaker: Dr Riccardo Di Maria (CERN)

ESCAPE Data Lake as a Service - 4th Rucio Community Workshop.pdf

Recording

17:15 XENON

Speaker: Paschalis Paschos (University of Chicago)

Data management with Rucio for the XENON experiment

Recording

17:30 GO/QoS/MAS

Speaker: Matt Snyder (Brookhaven National Laboratory)

Recording

RucioCommunityWorkshop2021_MattSnyder.odp

RucioCommunityWorkshop2021_MattSnyder.pdf

18:00 → 19:00 Panels: Astronomy & Metadata

Convener: Rosie Bolton (SKA Organisation)

18:00 Astronomy & Metadata

Speakers: Cedric Serfon (Brookhaven National Laboratory (US)), Dave Morris, Greg Daues (NCSA), Pandey Vishambhar (ASTRON), Rob Barnsley (SKAO)

Thursday, 30 September

15:00 → 16:00 Community Reports: Long tail of science

Convener: Alastair Dewhurst (Science and Technology Facilities Council STFC (GB))

15:00 Introduction - What is the 'Long Tail of Science'?

Speaker: Alastair Dewhurst (Science and Technology Facilities Council STFC (GB))

Recording

RucioWorkshop20210930.pdf

RucioWorkshop20210930.pptx

15:10 Multi-VO Rucio

Speaker: Timothy John Noble (Science and Technology Facilities Council STFC (GB))

Recording

RucioWorkshop_20210930.pdf

RucioWorkshop_20210930.pptx

15:25 Europe view (EGI)

Speaker: Andrea Manzi

Recording

15:40 Fermilab view

Speaker: Brandon White (Fermi National Accelerator Lab. (US))

Multi-VO Rucio at Fermilab.pdf

Recording

16:00 → 16:10 Break

16:10 → 17:10 Panels: Transfer & Storage

Convener: Mario Lassnig (CERN)

16:10 Transfer & Storage panel

Speakers: Andrea Ceccanti (Universita e INFN, Bologna (IT)), Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Hannah Short (CERN), Martin Barisits (CERN), Mihai Patrascoiu (CERN), Paul Millar

17:10 → 18:00 Opening & Closing: Discussion, Photo and Closing

Convener: Martin Barisits (CERN)

17:10 Discussion

17:40 Photo

17:45 Closing

Closing.pdf