Speaker
Description
For over a decade, the CERN IT Data Centres have been using a centralized monitoring infrastructure collecting data from hardware, services and applications via in-house sensors, metrics and notifications. Meanwhile also the LHC experiments were relying on dedicated WLCG Dashboards visualizing and reporting the status and progress of the job execution, data transfers and sites availability across the WLCG grid resources.
At the beginning of 2016 it was decided to merge services, resources and technologies of the two monitoring activities and move from in-house dedicated development toward open sources systems. This merge resulted in the definition and the development of a Unified Monitoring Architecture to collect, transport, store, search and visualize both IT Data Centres and WLCG Dashboard monitoring data. The newly developed architecture relies on state-of-the-art open source technologies and on open data formats, and provides solutions for easily collecting, processing and visualizing new monitoring data.
This contribution provides an overview of the Unified Monitoring Architecture, currently based on technologies such as collectd, ElasticSearch, Spark and Hadoop, with details on the lessons learned and on the ongoing work to monitor both the CERN IT Data Centres and the WLCG job, data transfers and sites and services. And, given the move to established open source technologies, it could also be easier to share experience and common solutions within the HEPiX community.
Scheduling constraints / preferences
Dear All, I will be presenting this talk and a talk about HammerCloud in the "Computing & Batch Services" track, it would be really great if the talks were not scheduled for the same time slot. Many thanks in advance! Jarka Schovancova (CERN IT)
| Length of talk (minutes) | 20 |
|---|
