15–19 Mar 2021
Europe/Zurich timezone

Anomaly Detection in the CERN Cloud Infrastructure

18 Mar 2021, 16:25
25m
Online workshop

Online workshop

Grid, Cloud & Virtualisation Grid, Cloud & Virtualisation

Speaker

Domenico Giordano (CERN)

Description

Anomaly Detection in the CERN Openstack Cloud is a challenging task due to the large scale of the computing infrastructure and the large volume of data to monitor.

The current solution to spot anomalous server machines in the cloud infrastructure relies on a threshold-based alarming system carefully set by the system managers on the performance metrics of each infrastructure component. The goal of this work is to explore fully automated and unsupervised machine learning solutions in the Anomaly Detection field. We exploit the current state-of-the-art solutions, including both traditional Anomaly Detection and Deep Anomaly Detection approaches.

This contribution will firstly describe the end-to-end data analytics pipeline that has been implemented to digest the large amount of monitoring data and expose anomalies to the system managers. The pipeline uses open source tools and frameworks, such as Spark, Apache Airflow, Kubernetes, Grafana, Elasticsearch. In addition, the performance of the aforementioned Anomaly Detection algorithms will be discussed.

Speaker release Yes

Primary author

Presentation materials