| Talk | |||||||||||||||||||||||
| Title | Next generation WLCG File Transfer Service (FTS) | ||||||||||||||||||||||
| Video | Loading
| ||||||||||||||||||||||
| Author(s) | Molnár, Zsolt (speaker) (CERN) | ||||||||||||||||||||||
| Corporate author(s) | CERN. Geneva | ||||||||||||||||||||||
| Imprint | 2012-05-24. - Streaming video, 00:26:46:00. | ||||||||||||||||||||||
| Series | (Conferences) (Computing in High Energy and Nuclear Physics (CHEP) 2012) | ||||||||||||||||||||||
| Lecture note | on 2012-05-24T17:00:00 | ||||||||||||||||||||||
| Subject category | Conferences | ||||||||||||||||||||||
| Abstract | LHC experiments at CERN and worldwide utilize WLCG resources and middleware components to perform distributed computing tasks. One of the most important tasks is reliable file replication. It is a complex problem, suffering from transfer failures, disconnections, transfer duplication, server and network overload, differences in storage systems, etc. To address these problems, EMI and gLite have provided the independent File Transfer Service (FTS) and Grid File Access Library (GFAL) tools. Their development started almost a decade ago, in the meantime, requirements in data management have changed - the old architecture of FTS and GFAL cannot keep support easily these changes. Technology has also been progressing: FTS and GFAL do not fit into the new paradigms (cloud, messaging, for example). To be able to serve the next stage of LHC data collecting (from 2013), we need a new generation of these tools: FTS 3 and GFAL 2. We envision a service requiring minimal configuration, which can dynamically adapt to the state of its resources (endpoints (SE-s) and network) and which offers an SE-centric configuration model for administrators to use where necessary. Main problems that we solve: scalability problems of the static channel model, resource and network states are not taken into account when scheduling transfer jobs, supporting multiple database backends (Oracle, MySQL), multiple transfer and control protocols. The new system will be much easier to configure and manage. FTS 3 will provide transparent job submission and monitoring features based on messaging. We report on design and prototype experience. We provide an overview about how FTS fits into the more general suite of data management utilities provided by the EMI project. | ||||||||||||||||||||||
| Copyright/License | © 2012-2024 CERN | ||||||||||||||||||||||
| Submitted by | jd@bnl.gov | ||||||||||||||||||||||
Record created 2012-07-10, last modified 2022-11-02
Talk details