4th Rucio Community Workshop (Virtual)

28 Sept 2021, 15:00 → 30 Sept 2021, 18:00 Europe/Zurich

Rucio is a software framework that provides functionality to organize, manage, and access large volumes of scientific data using customisable policies. The data can be spread across globally distributed locations and across heterogeneous data centers, uniting different storage and network technologies as a single federated entity. Rucio offers advanced features such as distributed data recovery or adaptive replication, and is highly scalable, modular, and extensible. Rucio has been originally developed to meet the requirements of the high-energy physics experiment ATLAS, and is continuously extended to support LHC experiments and other diverse scientific communities.

We set up a mailing list to which you can subscribe and where we will send more details about the program in the coming weeks.

We also created a Slack channel dedicated to the workshop discussion on the Rucio Slack workspace (Invitation Link). Join #workshop.

Group Photo.png

Tuesday 28 September

Opening & Closing

1 Welcome

Speaker: Martin Barisits (CERN)

Recording

Welcome 2021.pdf

2 Agenda & Logistics

Speaker: Mario Lassnig (CERN)

Agenda & Logistics.pdf

Recording

3 Rucio: State of the Union

Speaker: Martin Barisits (CERN)

Recording

Rucio State of the Union.pdf

Community Reports

Convener: Alastair Dewhurst (Science and Technology Facilities Council STFC (GB))

4 CMS

Speaker: Eric Vaandering (Fermi National Accelerator Lab. (US))

CMS Rucio WS 2021.pdf

Recording

5 Rucio for the Light Dark Matter eXperiment (LDMX)

The Light Dark Matter eXperiment (LDMX) is a planned small-scale accelerator-based experiment to search for dark matter in the sub-GeV mass region. Finalizing the design of the detector relies on Monte-Carlo simulation of expected physics processes. A distributed computing pilot project was initiated based around existing software used by other communities, including Rucio for data management. Rucio is primarily used as a dataset catalog, and LDMX also makes extensive use of Rucio metadata to record physics properties as well as bookkeeping information on the data produced. In this talk we describe how Rucio is used by LDMX and propose some possible extensions to the metadata functionality.

Speaker: David Cameron (University of Oslo (NO))

Recording

Rucio for LDMX.pdf

6 DUNE Rucio Usage and Status

We present the current state of the DUNE Rucio deployment and plans for extension and expansion. Results will include new monitoring and testing that has been
added as well as plans for moving from the current hybrid system to a native
Rucio deployment.

Speaker: Steven Timm (Fermi National Accelerator Lab. (US))

Recording

rucio_wkshp_2021sep28.pdf

rucio_wkshp_2021sep28.pptx

16:30 Break

Community Reports

Convener: Mario Lassnig (CERN)

7 Customizing Rucio at LCLS

This talk will present the speaker's onboarding experience as someone new to Rucio, covering the aspects of using documentation, building and standing up the containers, and configuring the Rucio system. Furthermore, specifics will be discussed in regard to the customization in development to meet the project requirements for LCLS, a free electron laser that produces ultra fast X-ray pulses.

Speaker: Kenny Lo (SLAC National Accelerator Laboratory)

LCLS_4thRucioWorkshop.pdf

Recording

8 IGWN

Speaker: Gabriele Gaetano Fronze' (INFN Torino (IT) and LIGO-Virgo-Kagra Collaboration (US/IT/JP))

Recording

RucioWorkshop2021-min.pdf

9 FTS: Updates, Direction and Plans

Speaker: Mihai Patrascoiu (CERN)

FTS_Rucio_Workshop_2021.pdf

Recording

10 AAI/Tokens IAM

Speaker: Andrea Ceccanti (Universita e INFN, Bologna (IT))

20210928-IAM-Rucio-Workshop.pdf

Recording

Panels: Rucio in a non-grid environment

Convener: Cedric Serfon (Brookhaven National Laboratory (US))

11 Rucio in a non-grid environment panel

Speakers: Alastair Dewhurst (Science and Technology Facilities Council STFC (GB)), Andrea Manzi, David Cameron (University of Oslo (NO)), Ilija Vukotic (University of Chicago (US)), Mario Lassnig (CERN), Oliver Keeble (CERN)

Wednesday 29 September

Keynote

Convener: Mario Lassnig (CERN)

12 Pipe Dreams (and Nightmares)

Australia's Academic and Research Network (AARNet) was established in 1989 and is widely regarded as the founder of the Internet in Australia and renowned as the architect, builder and operator of world-class network infrastructure for research and education.
We are Australia's National Research and Education Network (NREN). We connect over one million users—researchers, faculty, staff and students—at institutions across Australia, supporting education and research across a diverse range of disciplines including high energy physics, climate science, genomics, radio astronomy and the arts.
Nationally, AARNet interconnects Australian universities, the CSIRO, and other organisations who have a research and education mission, or with whom the education and research sector interacts. These include hospitals, vocational training providers, schools and museums. Internationally, AARNet interconnects the Australian Research and Education (R & E) community to the world – and continuously develops new capabilities and partnerships to facilitate seamless data access and transfer.
Today, we'll talk about some of the work we've done in the data access and transfer space, challenges we've faced, and our plans for the future.

Speaker: Crystal Michelle Chua

cc_ga_rucio4.pdf

Recording

Community Reports: WFMS

Convener: Cedric Serfon (Brookhaven National Laboratory (US))

13 Panda

Speaker: Paul Nilsson (Brookhaven National Laboratory (US))

PanDA and Rucio, September 29, 2021.pdf

Recording

14 DIRAC

Speaker: Federico Stagni (CERN)

Recording

RucioWS.pdf

15 CMS Workload Management and Rucio integration

Speaker: Todor Trendafilov Ivanov (University of Sofia - St. Kliment Ohridski (BG))

Recording

Rucio at WMCore.pdf

16 Discussion

11:05 Break

Community Reports

Convener: Martin Barisits (CERN)

17 Belle II

Speaker: Cedric Serfon (Brookhaven National Laboratory (US))

2021-09-29 - Rucio and Belle II.pdf

Recording

18 Rucio and ScienceMesh: Enabling data management for the CS3 community

Speakers: Giuseppe Lo Presti (CERN), Rahul Chauhan

Chauhan, LoPresti - ScienceMesh - Rucio workshop 2021.pdf

Recording

19 ATLAS

Speaker: Mario Lassnig (CERN)

Recording

Rucio @ ATLAS.pdf

Community Reports: Astronomy

Convener: Rosie Bolton (SKA Organisation)

20 Session Introduction

Speaker: Rosie Bolton (SKA Organisation)

Recording

21 SKA Rucio deployment and metadata/ data lifecycle use case

Speaker: Rohini Joshi (SKA Organisation)

Recording

SKAO - Rucio Community Workshop.pdf

22 MAGIC telescope rucio deployment

Speaker: Agustin Bruzzese

ESCAPE-2021-09-29.pdf

Recording

23 CTA rucio use case and development with cloud storage

Speaker: Frederic Gillardo (Centre National de la Recherche Scientifique (FR))

Recording

RucioUseCase-2021.pdf

RucioUseCase-2021.pptx

24 LOFAR Use Cases in ESCAPE - Experience and Future Directions

We will be presenting our experiences and results for LOFAR use case using ESCAPE data lake and Rucio. Reflections on future requirements including for other data sets (e.g. APERTIF) shall also be deliberated.

Speaker: Yan Grange (ASTRON)

Recording

Rucio for LOFAR.pdf

25 IVOA

Speaker: Dave Morris

20210929-ESCAPE-Rucio.pdf

Recording

26 Rucio @ SDC/KIS

Speaker: Peter Caligari (KIS)

2021_09_4thRucioWS21_PC.pdf

Recording

16:40 Break

Community Reports

Convener: Eric Vaandering (Fermi National Accelerator Lab. (US))

27 ESCAPE Data Lake as a Service

Experiments and scientists, whether in the process of designing and building up a data management system or managing multi-petabyte data historically, gather in the European Science Cluster of Astronomy & Particle physics ESFRI research infrastructures (ESCAPE) project to address computing challenges by developing common solutions in the context of the EOSC.
A modular ecosystem of services and tools constitutes the ESCAPE Data Lake, which is exploited by flagship ESFRIs in Astro-particle Physics, Electromagnetic and Gravitational-Wave Astronomy, Particle Physics, and Nuclear Physics to pursue together the FAIR and open-access data principles.
This infrastructure fulfils the needs of the ESCAPE community in terms of data organisation, management, and access, and dedicated assessment exercises demonstrated its robustness.
As a result, collaborating sciences are choosing their reference implementations of the various technologies among the proposed solutions.
A variety of challenges and specific use cases boost ESCAPE to carefully take into account both user and infrastructure perspectives, and contributed to successfully conclude the pilot phase beyond expectations, embarking on a like-production prototype stage.
The ongoing phase of the project aims at consolidating the functionalities of the services, e.g. integrating token-based AuthN/Z or deploying a tailored content delivery and caching layer, and at simplifying the user experience. Specifically for this reason, a considerable effort is being devoted towards a DataLake-as-a-Service whose goal is to provide the end-user with a Notebook ready-to-be-used and fully integrated with the Data Lake.
ESCAPE milestones achieved during the length of the project represent a fundamental accomplishment under both sociological and computing model aspects for different scientific communities that should address upcoming data management and computing challenges in the next decade.

Speaker: Dr Riccardo Di Maria (CERN)

ESCAPE Data Lake as a Service - 4th Rucio Community Workshop.pdf

Recording

28 XENON

Speaker: Paschalis Paschos (University of Chicago)

Data management with Rucio for the XENON experiment

Recording

29 GO/QoS/MAS

Speaker: Matt Snyder (Brookhaven National Laboratory)

Recording

RucioCommunityWorkshop2021_MattSnyder.odp

RucioCommunityWorkshop2021_MattSnyder.pdf

Panels: Astronomy & Metadata

Convener: Rosie Bolton (SKA Organisation)

30 Astronomy & Metadata

Speakers: Cedric Serfon (Brookhaven National Laboratory (US)), Dave Morris, Greg Daues (NCSA), Pandey Vishambhar (ASTRON), Rob Barnsley (SKAO)

Thursday 30 September

Community Reports: Long tail of science

Convener: Alastair Dewhurst (Science and Technology Facilities Council STFC (GB))

31 Introduction - What is the 'Long Tail of Science'?

Speaker: Alastair Dewhurst (Science and Technology Facilities Council STFC (GB))

Recording

RucioWorkshop20210930.pdf

RucioWorkshop20210930.pptx

32 Multi-VO Rucio

Speaker: Timothy John Noble (Science and Technology Facilities Council STFC (GB))

Recording

RucioWorkshop_20210930.pdf

RucioWorkshop_20210930.pptx

33 Europe view (EGI)

Speaker: Andrea Manzi

Recording

34 Fermilab view

Speaker: Brandon White (Fermi National Accelerator Lab. (US))

Multi-VO Rucio at Fermilab.pdf

Recording

16:00 Break

Panels: Transfer & Storage

Convener: Mario Lassnig (CERN)

35 Transfer & Storage panel

Speakers: Andrea Ceccanti (Universita e INFN, Bologna (IT)), Andrew Bohdan Hanushevsky (SLAC National Accelerator Laboratory (US)), Hannah Short (CERN), Martin Barisits (CERN), Mihai Patrascoiu (CERN), Paul Millar

Opening & Closing: Discussion, Photo and Closing

Convener: Martin Barisits (CERN)

36 Discussion

37 Photo

38 Closing

Closing.pdf