13–17 Oct 2025
Hotel Exe Campus
Europe/Madrid timezone

Benchmarking the Open Science Data Federation services - S3 plugin and cache selection

16 Oct 2025, 16:40
20m
Lluis Vives I (Hotel Exe Campus)

Lluis Vives I

Hotel Exe Campus

Bellaterra (Barcelona), Spain

Speaker

Fabio Andrijauskas (Univ. of California San Diego (US))

Description

Research has become dependent on processing power and storage, one crucial aspect being data sharing. The Open Science Data Federation (OSDF) project aims to create a scientific global data distribution network based on the Pelican Platform. OSDF does not develop new software but relies on the XrootD and Pelican projects. Nevertheless, OSDF must understand the XrootD limits under various configuration options, including transfer rate limits, proper buffer configuration, and storage type effect. This work describes the tests and results performed using National Research Platform (NRP) hosts, showing the S3 plugin and cache selection process. The tests cover various file sizes and parallel streams and use clients from various distances from the server host. We also used several standalone clients (wget, curl, pelican) and the native HTCondor file transfer mechanisms.

Author

Fabio Andrijauskas (Univ. of California San Diego (US))

Presentation materials