Speaker
Michail Salichos
(CERN)
Description
FTS is the service responsible for distributing the majority of LHC data across the WLCG infrastructure. From the experiences of the last decade supporting and monitoring FTS, reliability, robustness and
high-performance data transfers has proved to be of high importance in the Data Management world. We are going to present the current status and features of the new File Transfer Service (FTS3), which address the problems that the previous FTS version face with: static channel model, configuration and scalability problems, new protocols support, more database back-ends support, etc. We present the solution we proposed and the design of the new tools as well the reliability, stability, scalability and performance requirements of a data movement middle-ware in the grid environment. The service has already undergone extensive pre-production validation and we report the results of high volume production transfers performed on the pilot service.
Anticipating the upcoming data movement needs of WLCG, and building on the lessons learned during the first run, we present a new, scalable and highly-optimized data movement service, which provides a simple interface for transfer job submission, status retrieval, advanced monitoring capabilities, multiple access and transfer protocols support and simplified configuration.
Transfer auto-tuning (dynamically adjusting the number of active
transfers based on success/failure rate and achieved throughput),
endpoint-centric VO share configuration, multiple replicas support,
REST-style interface for transfer submission and status retrieval,
staging files from archive, support for Oracle and MySQL database
back-ends, multiple transfer and access protocols support using gfal2
plug-in mechanism (namely SRM, gsiftp, http and xroot are already
implemented) and session/connection reuse (gridftp, ssl, etc), are only some of the new features and functionality that FTS3 has been delivered with. In order to be a credible long-term platform for data transfer, FTS3 has been designed to exploit upcoming developments in networking, such as integrating monitoring data from perfsonar for further transfer optimization, resource management and monitoring network state.
FTS3 aims to become the new data movement service for the WLCG
infrastructure.
Primary author
Michail Salichos
(CERN)
