21-25 May 2012
New York City, NY, USA
US/Eastern timezone

Application of rule based data mining techniques to real time ATLAS Grid job monitoring data

22 May 2012, 13:30
4h 45m
Rosenthal Pavilion (10th floor) (Kimmel Center)

Rosenthal Pavilion (10th floor)

Kimmel Center

Poster Distributed Processing and Analysis on Grids and Clouds (track 3) Poster Session

Speaker

Sergey Kalinin (Bergische Universitaet Wuppertal (DE))

Description

The Job Execution Monitor (JEM), a job-centric grid job monitoring software, is actively developed at the University of Wuppertal. It leverages Grid-based physics analysis and Monte Carlo event production for the ATLAS experiment by monitoring job progress and grid worker node health. Using message passing techniques, the gathered data can be supervised in real time by users, site admins and shift personnel. Imminent error conditions can be detected early and countermeasures taken by the Job's owner. Grid site admins can access aggregated data of all monitored jobs to infer the site status and to detect job and Grid worker node misbehavior. Shifters can use the same aggregated data to quickly react to site error conditions and broken production jobs. JEM is integrated into ATLAS' Pilot-based "PanDA" job brokerage system. In this work, the application of novel data-centric rule based methods and data-mining techniques to the real time monitoring data is discussed. The usage of such automatic inference techniques on monitoring data to provide job- and site-health summary information to users and admins is presented. Finally, the provision of a secure real-time control- and steering channel to the job as extension of the presented monitoring software is considered and a possible architecture is shown

Primary author

Co-authors

Frank Volkmer (Fachbereich C / Physik-Bergische Universitaet Wuppertal-Unknown) Marisa Sandhoff (Bergische Universitaet Wuppertal (DE)) Raphael Ahrens Sergey Kalinin (Bergische Universitaet Wuppertal (DE)) Tim Dos Santos (Bergische Universitaet Wuppertal (DE)) Mr Torsten Harenberg (UNIVERSITY OF WUPPERTAL)

Presentation Materials