27 September 2004 to 1 October 2004
Interlaken, Switzerland
Europe/Zurich timezone

Software agents in data and workflow management

29 Sep 2004, 15:40
Theatersaal (Interlaken, Switzerland)


Interlaken, Switzerland

oral presentation Track 4 - Distributed Computing Services Distributed Computing Services




CMS currently uses a number of tools to transfer data which, taken together, form the basis of a heterogenous datagrid. The range of tools used, and the directed, rather than optimised nature of CMS recent large scale data challenge required the creation of a simple infrastructure that allowed a range of tools to operate in a complementary way. The system created comprises a hierarchy of simple processes (named agents) that propagate files through a number of transfer states. File locations and some application metadata were stored in POOL file catalogues, with LCG LRC or MySQL backends. Agents were assigned limited responsibilities, and were restricted to communicating state in a well-defined, indirect fashion through a central transfer management database. In this way, the task of distributing data was easily divided between different groups for implementation. The prototype system was developed rapidly, and achieved the required sustained transfer rate of ~10 MBps, with O(10^6) files distributed to 6 sites from CERN. Experience with the system during the data challenge raised issues with underlying technology (MSS write/read, stability of the LRC, maintenance of file catalogues, synchronisation of filespaces _) which have been successfully identified and handled. The development of this prototype infrastructure allows us to plan the evolution of backbone CMS data distribution from a simple hierarchy to a more autonomous, scalable model drawing on emerging agent and grid technology.

Primary authors

A. Afaq (FNAL) A. Fanfani (INFN-Bologna) C. Charlot (IN2P3, LLR) C. Grandi (INFN-Bologna) D. Bonacorsi (CNAF-INFN) D. Newbold (University of Bristol) I. Fisk (FNAL) J. Hernandez (CIEMAT Madrid) J. Rehn (University of Karlsruhe) K. Rabbertz (University of Karlsruhe) L. Tuura (Northeastern University) M. Ernst (FNAL) O. Maroney (University of Bristol) P. GARCIA-ABIA (CIEMAT) S. Metson (University of Bristol) T. Barrass (CMS, UNIVERSITY OF BRISTOL) T. Wildish (Princeton) W. Jank (CERN) Y. Wu (FNAL)

Presentation Materials