27 September 2004 to 1 October 2004
Interlaken, Switzerland
Europe/Zurich timezone

Application of the SAMGrid Test-Harness for Performance Evaluation and Tuning of a Distributed Cluster Implementation of Data Handling Services

27 Sep 2004, 17:50
20m
Ballsaal (Interlaken, Switzerland)

Ballsaal

Interlaken, Switzerland

oral presentation Track 5 - Distributed Computing Systems and Experiences Distributed Computing Systems and Experiences

Speaker

A. Lyon (FERMI NATIONAL ACCELERATOR LABORATORY)

Description

The SAMGrid team has recently refactored its test harness suite for greater flexibility and easier configuration. This makes possible more interesting applications of the test harness, for component tests, integration tests, and stress tests. We report on the architecture of the test harness and its recent application to stress tests of a new analysis cluster at Fermilab, to explore the extremes of analysis use cases and the relevant parameters for tuning in the SAMGrid station services. This reimplementation of the test harness is a python framework which usesXML for configuration and small plug-in python modules for specific test purposes. One current testing application is running on a 128-CPU analysis cluster with access to 6 TB distributed cache and also to a 2 TB centralized cache, permitting studies of different cache strategies. We have studied the service parameters which affect the performance of retrieving data from tape storage as well. The use cases studied vary from those which will require rapid file delivery with short processing time per file, to the opposite extreme of long processing time per file. We also show how the same harness can be used to run regular unit tests on a production system to aid early fault detection and diagnosis.These results are interesting for their implications with regard to Grid operations, and illustrate the type of monitoring and test facilities required to accomplish such performance tuning.

Primary authors

A. Baranovski (FERMI NATIONAL ACCELERATOR LABORATORY) A. Kreymer (FERMI NATIONAL ACCELERATOR LABORATORY) A. Lyon (FERMI NATIONAL ACCELERATOR LABORATORY) A. Sill (Texas Tech University) F. Ratnikov (Rutgers University) G. Garzoglio (FERMI NATIONAL ACCELERATOR LABORATORY) I. Terekhov (FERMI NATIONAL ACCELERATOR LABORATORY) J. Trumbo (FERMI NATIONAL ACCELERATOR LABORATORY) L. Loebel Carpenter (Fermilab) L. Lueking (FERMI NATIONAL ACCELERATOR LABORATORY) M. Burgon-Lyon (Glasgow University) M. Leslie (Oxford University) R. Herber (FERMI NATIONAL ACCELERATOR LABORATORY) R. Illingworth (FERMI NATIONAL ACCELERATOR LABORATORY) R. Kennedy (FERMI NATIONAL ACCELERATOR LABORATORY) R. St.Denis (Glasgow University) S. Belforte (INFN/Trieste) S. Stonjek (FERMI NATIONAL ACCELERATOR LABORATORY/Oxford University) S. White (FERMI NATIONAL ACCELERATOR LABORATORY) U. Kerzel (Karlsruhe University) V. Bartsch (Oxford University) W. Merritt (FERMI NATIONAL ACCELERATOR LABORATORY)

Presentation Materials