Speaker
Description
Large-scale scientific collaborations such as WLCG need reliable and secure data transfers that optimize the available bandwidth and resources of the grid. HTTP-based third-party copy (TPC) transfers follow a de-facto community standard for moving files directly between storage endpoints (peer-to-peer). Here we report on an extension to that standard promoting improved data integrity through implementation of the IETF RFC 3230 (Instance Digest) standard for end-to-end checksum verification. The protocol is designed to integrate transparently with existing TPC workflows, enabling automatic digest negotiation and validation without affecting current operations.
Early implementations of this updated protocol include storage backends such as EOS and CERN Tape Archive (CTA) and transfer orchestration via FTS. Adopting this new standard introduces its own challenges, such as computing checksums on the fly during large-scale transfers and ensuring consistent validation across heterogeneous storage systems. At the same time, it creates opportunities, including stronger token-based security, improved verification mechanisms, and better reproducibility of distributed workflows.
Finally, in this work we explore the lifecycle of this new development, from the original idea to ongoing work and to future plans to settle it as the default data integrity mechanism for LHC Run 4.