Nov 4 – 8, 2019
Adelaide Convention Centre
Australia/Adelaide timezone

Erratic server behavior detection using machine learning on streams of monitoring data

Nov 5, 2019, 3:30 PM
1h
Hall F (Adelaide Convention Centre)

Hall F

Adelaide Convention Centre

Poster Track 7 – Facilities, Clouds and Containers Posters

Speaker

Martin Adam (Acad. of Sciences of the Czech Rep. (CZ))

Description

With the explosion of the number of distributed applications, a new dynamic server environment emerged grouping servers into clusters, utilization of which depends on the current demand for the application. To provide reliable and smooth services it is crucial to detect and fix possible erratic behavior of individual servers in these clusters. Use of standard techniques for this purpose requires manual work and delivers suboptimal results.

Using only application agnostic monitoring metrics our machine learning based method analyzes the recent performance of the inspected server as well as the state of the rest of the cluster, thus checking not only the behavior of the single server, but the load on the whole distributed application as well. We have implemented our method in a Spark job running in the CERN MONIT infrastructure.

In this contribution we present results of testing multiple machine learning algorithms and preprocessing techniques to identify the servers erratic behavior. We also discuss the challenges of deploying our new method into production.

Consider for promotion No

Primary authors

Martin Adam (Acad. of Sciences of the Czech Rep. (CZ)) Luca Magnoni (CERN)

Co-authors

Dagmar Adamova (Acad. of Sciences of the Czech Rep. (CZ)) Mr Martin Pilát (Faculty of Mathematics and Physics, Charles University)

Presentation materials