Nov 4 – 8, 2019
Adelaide Convention Centre
Australia/Adelaide timezone

Modernizing Third-Party-Copy Transfers in WLCG

Nov 5, 2019, 11:00 AM
Riverbank R8 (Adelaide Convention Centre)

Riverbank R8

Adelaide Convention Centre

Oral Track 4 – Data Organisation, Management and Access Track 4 – Data Organisation, Management and Access


Alessandra Forti (University of Manchester (GB))


The “Third Party Copy” (TPC) Working Group in the WLCG’s “Data Organization, Management, and Access” (DOMA) activity was proposed during a CHEP 2018 Birds of a Feather session in order to help organize the work toward developing alternatives to the GridFTP protocol. Alternate protocols enable the community to diversify; explore new approaches such as alternate authorization mechanisms; and reduce the risk due to the retirement of the Globus Toolkit, which provides a commonly used GridFTP protocol implementation.

Two alternatives were proposed to the TPC group for investigation: WebDAV and XRootD. Each approach has multiple implementations, allowing us to demonstrate interoperability between distinct storage systems. As the working group took as a mandate the development of alternatives - and not to select a single protocol - we have put together a program of work allowing both to flourish. This includes community infrastructure such as documentation pointers, email lists, or biweekly meetings, as well as continuous interoperability testing involving production & test endpoints, deployment recipes, scale testing, and debugging assistance.

Each major storage system utilized by WLCG sites now has at least one functional non-GridFTP protocol for performing third-party-copy. The working group is focusing on including a wider set of sites and helping sites deploy more production endpoints. We are interacting with WLCG VOs to perform production data transfers using WebDAV or XRootD at the participating sites with the objective that all sites deploy at least one of these alternative protocols.

Consider for promotion Yes

Primary authors

Thomas Beermann (University of Innsbruck (AT)) Brian Paul Bockelman (University of Nebraska Lincoln (US)) Andrea Ceccanti (Universita e INFN, Bologna (IT)) Alessandra Forti (University of Manchester (GB)) Fabrizio Furano (CERN) Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER) Mario Lassnig (CERN) Andrea Manzi (CERN) Paul Millar Horst Severini (University of Oklahoma (US)) Dr Albert Rossi (FNAL)

Presentation materials