Hassen Riahi & Daniele Spiga, AsyncStageOut
Abstact:
As part of the new generation of CMS Workload Management tools, the Asynchronous Stage-Out system (AsyncStageOut) has been developed to enable third party copy of the user output. The AsyncStageOut component manages glite FTS transfers of data from the temporary store at the site where the job ran to the final location of the data on behalf of that data owner. The tool uses python daemons, built using the WMCore framework. It is designed as a thin application relying only on the NoSQL database (CouchDB) as input, data storage and platform for a dedicated operations monitoring system. It has progressed from a limited prototype to a highly adaptable service which manages and monitors the whole user files steps, namely file transfer and publication.
The AsyncStageOut foresees the management of nearly 200k users' files per day of close to 1000 users per month with minimal delays, and providing a real time monitoring and reports to users and service operators, while being highly available.