Hassen Riahi & Daniele Spiga, AsyncStageOut

→ Europe/Zurich
28/S-029 (CERN)

28/S-029

CERN

35
Show room on map
Description
WhiteArea lectures' twiki HERE

Abstact:

The CMS distributed data analysis workflow assumes that jobs run in a different location from where their results are finally stored. Typically the user output must be transferred across the network from one site to another, possibly on a different continent. This step is named stage-out and in CMS was originally implemented as a synchronous step of the analysis job execution. However, our experience showed the weakness of this approach both in terms of low total job execution efficiency and failure rates, wasting precious CPU resources.

As part of the new generation of CMS Workload Management tools, the Asynchronous Stage-Out system (AsyncStageOut) has been developed to enable third party copy of the user output. The AsyncStageOut component manages glite FTS transfers of data from the temporary store at the site where the job ran to the final location of the data on behalf of that data owner. The tool uses python daemons, built using the WMCore framework. It is designed as a thin application relying only on the NoSQL database (CouchDB) as input, data storage and platform for a dedicated operations monitoring system. It has progressed from a limited prototype to a highly adaptable service which manages and monitors the whole user files steps, namely file transfer and publication.

The AsyncStageOut foresees the management of nearly 200k users' files per day of close to 1000 users per month with minimal delays, and providing a real time monitoring and reports to users and service operators, while being highly available.

Slides
The agenda of this meeting is empty