Speakers
Description
Rucio, the scientific data management system developed by ATLAS at CERN, has become widely adopted across high-energy physics experiments for managing distributed datasets at exabyte scale. Traditionally, Rucio relies on the WLCG File Transfer Service (FTS) for data movement between storage elements. We present recent developments enabling Globus—the research cyberinfrastructure platform operated by the University of Chicago and used extensively by DOE-ASCR and NSF HPC facilities—as an alternative transfer tool within Rucio.
Building on initial work from 2019, we have updated and validated the Globus transfer tool integration with the latest Rucio releases. The implementation uses OAuth2 authentication with the Globus SDK, supports both bulk and single-file transfers, and works with Globus Server endpoints as well as Globus Connect Personal endpoints. The integration required only approximately 100 lines of code modifications to Rucio's transfer tool components.
We report on functional testing of transfers and deletions using test RSEs, and describe ongoing scale testing between Argonne’s LCRC and NERSC at LBNL. This integration provides HEP experiments with access to Globus's 10,000+ connected storage systems across 2,600+ institutions, while enabling the ASCR community to leverage Rucio's sophisticated data management capabilities. We discuss plans for adoption by DUNE, ATLAS, Rubin Observatory, and other experiments, as well as future work on Globus Connect Personal support for end-user data access.