Nov 4 – 8, 2019
Adelaide Convention Centre
Australia/Adelaide timezone

Using WLCG data management software to support other communities

Nov 4, 2019, 3:15 PM
Riverbank R8 (Adelaide Convention Centre)

Riverbank R8

Adelaide Convention Centre

Oral Track 4 – Data Organisation, Management and Access Track 4 – Data Organisation, Management and Access


Ian Collier (Science and Technology Facilities Council STFC (GB))


When the LHC started data taking in 2009 the data rates were unprecedented for the time and forced the WLCG community develop a range of tools for managing their data across many different sites. A decade later other science communities are finding their data requirements have grown far beyond what they can easily manage and are looking for help. The RAL Tier-1’s primary mission has always been to provide resources for the LHC experiments although 10% is set aside for non-LHC experiments. In the last 2 years the Tier-1 has received additional funding to support other scientific communities and now provides over 5 PB disk and 13PB Tape storage to them.

RAL has run an FTS service for the LHC experiments for several years. Smaller scientific communities have used this for moving large volumes of data between sites but have frequently reported difficulties. In the past RAL also provided an LFC service for managing files stored on the Grid. The RAL Tier-1 should provide a complete data management solution for these communities and it was therefore decided to setup a Rucio instance to do this.

In April 2018 a Rucio instance was setup at RAL which has so far been used by the AENEAS project. RAL is providing the development effort to allow a single Rucio instance to support multiple experiments. RAL also runs a DynaFed service which is providing an authentication and authorization layer in front of RAL S3 storage service.

Consider for promotion No

Primary authors

Alastair Dewhurst (Science and Technology Facilities Council STFC (GB)) Catalin Condurache (Science and Technology Facilities Council STFC (GB)) Tom Byrne (STFC) Mr Ian Johnson (Science and Technology Facilities Council STFC (GB))

Presentation materials