Choose timezone
Your profile timezone:
JCOP FWWG 31 Oct 2017: Elasticsearch for logs
→
Europe/Zurich
40/R-C10 (CERN)
Use of the Elastic Stack for WinCC OA log analysis.
No news on WinCC OA; work progressing on JCOP Framework the release of which is expected on the first week of December.
James presents the log analysis with Elasticsearch; it is an update to the presentation from JCOP FWWG in November 2016, https://indico.cern.ch/event/589033/ ; there were a few important changes since then.
IT is providing the Elasticsearch infrastructure and maintain/tune it; one needs to provide them the estimates of how much space per month we'd need. BE/ICS has over 140 projects running since the beginning of the year, with around 2-3 mln log entries per day.
Messages come with less than 30 seconds delay, under nominal conditions (typically few seconds). There is also a possibility to have "live" view, with updates every few seconds.
Another use case for Elasticsearch in BE-ICS applications is to calculate statistics of Oracle Archiver (how many value changes per day per device, how many alarms). Over 100 projects covered.
Lukasz presented a couple of dashboards/filters that he uses on daily basis as diagnostic/monitoring tool for the BE/ICS SCADA Application Service. One could report, for instance, on syntax errors and find the panels or libraries causing most errors and showing in which line. This is augmented by live-filtering to drill down.
Up to now only one incident with Logstash/FileBeats: there was a chance of running of disk space on broken connection, yet we did not really investigated what happened. There are settings to moderate this behaviour and tune the filebeats component.
The query language of ES is not very intuitive at the beginning, but very powerful.
We could also tap directly to ES to get the log messages and process them, without going to the project.
There are dashboards that monitor the service itself, with performance graphs, queue lengths, errors, etc.
Q: Stefan, How do we deploy this setup:
Filebeat installation, one big configuration file for all, copied to all WinCC OA servers. This file is actually generated from templates; everything automated with scripts; could be automatized further with RPMs.
Q: does this support pattern matches for file loctions?
yes, surely. However for BE-ICS appllications have different paths
Q: Does it work on Windows as well?
Yes, BE-ICS have one Windows project as well (filebeats runs there). Logstash is running on a openstack linux machine.
Q: How many machines do you need
A few, they are easily scaled up; hosted on openstack
Q: How you grant access to Kibana?
Based on egroups; IT set it up on BE-ICS instance of Kibana. Granularity is crude - people cannot change the dashboards or everybody could change their own. There are paid plugins for this. There are many other paid plugins available
Presentation of NA62:
- tried splunk before; gave up due to limitations of the non-paid version; switched to Elastic stack
- runs on a kind of shared "central" service. Frequent upgrades lead sometimes to unavailability of the service. Very positive feedback - will run in 2018 too.
Q/A: The service is not yet really declared as fully in production.
Q/A: What do you get from IT? They set up ES+Kibana service and give REST endpoints; one needs to set up filebeats and logstash, Openstack seems to be OK for it.
Q/A: How would you drill down from the "peak" in the graph to the root cause. By clicking you put more and more selective filters.
Comment from CMS: statistical log analysis should not replace powerful alarm systems. Acknowledged, with a notice that some of WinCC OA diagnostic information is only available through logs.
Q/A: Would it be possible to decrease the disk usage (disk buffering). There are some parameters to parameterize the throughput/behaviour.
Q/A: Could this a possible replacement for the central logging component? To be analysed/decided later
ATLAS and ALICE interested in trying this out. CMS not convinced. NA62 already using it with success for they computing farms for this year and planning to extend in 2018 to other systems. BE-ICS running in a stable way since April. LHCb - to be confirmed
No AOB.
Next meeting tentatively scheduled in 2 weeks to present the state of preparation for the JCOP Framework release, discuss top outstanding issues.
There are minutes attached to this event.
Show them.