Nov 4 – 8, 2019
Adelaide Convention Centre
Australia/Adelaide timezone

The Dynafed data federator as grid site storage element

Nov 5, 2019, 5:00 PM
15m
Riverbank R8 (Adelaide Convention Centre)

Riverbank R8

Adelaide Convention Centre

Oral Track 4 – Data Organisation, Management and Access Track 4 – Data Organisation, Management and Access

Speaker

Frank Berghaus (University of Victoria (CA))

Description

The Dynafed data federator is designed to present a dynamic and unified view of a distributed file repository. We describe our use of Dynafed to construct a production-ready WLCG storage element (SE) using existing Grid storage endpoints as well as object storage. In particular, Dynafed is used as the primary SE for the Canadian distributed computing cloud systems. Specifically, we have been using a Dynafed-based SE in production for reading input files of Belle-II jobs. We have run up to 6000 Belle-II jobs simultaneously on distributed cloud resources, requiring the transfer of approximately 60 TB of input data per day. Similarly, we have been using a Dynafed-based SE in pre-production testing for the ATLAS experiment. We will describe the configuration of Dynafed to make it suitable as a WLCG SE. In particular, we will highlight the improvements within Dynafed that make it possible to do checksum based file verification and 3rd-party file copy via WebDAV. We will also report on a new monitoring system and an automated system that collects storage information from all endpoints. The monitoring is accessible via a web browser and also on the command line. It includes, for example, information of who requested files, which endpoints served the files, how often each file was requested from an endpoint, as well as endpoint storage information collected by the system we developed. The storage information system collects information about each storage endpoint which is then used by Dynafed to identify endpoints with sufficient storage capacity. The presentation will discuss the configuration and successful operation of Dynafed within the context of a distributed compute cloud infrastructure.

Consider for promotion Yes

Primary authors

Colin Roy Leavett-Brown (University of Victoria (CA)) Colson Driemel (University of Victoria) Fabrizio Furano (CERN) Fernando Fernandez Galindo (TRIUMF (CA)) Frank Berghaus (University of Victoria (CA)) Dr Kevin Casteels (University of Victoria) Marcus Ebert (University of Victoria) Michael Paterson (U) Oliver Keeble (CERN) Randy Sobie (University of Victoria (CA)) Reda Tafirout (TRIUMF (CA)) Rolf Seuster (University of Victoria (CA))

Presentation materials