14-18 October 2013
Amsterdam, Beurs van Berlage
Europe/Amsterdam timezone

FTS3 – Robust, simplified and high-performance data movement service for WLCG

14 Oct 2013, 17:25
22m
Keurzaal (Amsterdam, Beurs van Berlage)

Keurzaal

Amsterdam, Beurs van Berlage

Oral presentation to parallel session Distributed Processing and Data Handling B: Experiment Data Processing, Data Handling and Computing Models Distributed Processing and Data Handling B: Experiment Data Processing, Data Handling and Computing Models

Speaker

Michail Salichos (CERN)

Description

FTS is the service responsible for distributing the majority of LHC data across the WLCG infrastructure. From the experiences of the last decade supporting and monitoring FTS, reliability, robustness and high-performance data transfers has proved to be of high importance in the Data Management world. We are going to present the current status and features of the new File Transfer Service (FTS3), which address the problems that the previous FTS version face with: static channel model, configuration and scalability problems, new protocols support, more database back-ends support, etc. We present the solution we proposed and the design of the new tools as well the reliability, stability, scalability and performance requirements of a data movement middle-ware in the grid environment. The service has already undergone extensive pre-production validation and we report the results of high volume production transfers performed on the pilot service. Anticipating the upcoming data movement needs of WLCG, and building on the lessons learned during the first run, we present a new, scalable and highly-optimized data movement service, which provides a simple interface for transfer job submission, status retrieval, advanced monitoring capabilities, multiple access and transfer protocols support and simplified configuration. Transfer auto-tuning (dynamically adjusting the number of active transfers based on success/failure rate and achieved throughput), endpoint-centric VO share configuration, multiple replicas support, REST-style interface for transfer submission and status retrieval, staging files from archive, support for Oracle and MySQL database back-ends, multiple transfer and access protocols support using gfal2 plug-in mechanism (namely SRM, gsiftp, http and xroot are already implemented) and session/connection reuse (gridftp, ssl, etc), are only some of the new features and functionality that FTS3 has been delivered with. In order to be a credible long-term platform for data transfer, FTS3 has been designed to exploit upcoming developments in networking, such as integrating monitoring data from perfsonar for further transfer optimization, resource management and monitoring network state. FTS3 aims to become the new data movement service for the WLCG infrastructure.

Primary author

Co-authors

Presentation Materials