Centre de Calcul de l'Institut National de Physique Nucleaire et de Physique des Particules

Typical syslog-ng use-cases

CC-IN2P3

Fabien Wernli

Outline

                    talk {
                      infrastructure();
                      altertative { ... };
                      channel {
                        storing(); alerting(); enriching();
                      };
                      channel {
                        misc (); automation(); monitoring();
                      };
                      channel {
                        appendix { ... };
                        flags(if-time-permits);
                      };
                    };
          
Architecture

Architecture

Architecture
Storing Logs

Storing Logs

Elasticsearch

                destination d_elasticsearch {
                  elasticsearch2(
                    client-lib-dir("/usr/share/elasticsearch/plugins/search-guard-5/*.jar:/usr/share/elasticsearch/lib/")
                    client-mode("https")
                    concurrent-requests("16")
                    disk-buffer(
                      dir("/var/lib/syslog-ng-disq/")
                      disk-buf-size(53687091200)
                      mem-buf-size(1073741824)
                    )
                    flush-limit('1024')
                    index("${__es_index:-syslog}-${YEAR}.${MONTH}.${DAY}")
                    port('9200')
                    server("node01 node02 node03 node04 node05")
                    java_keystore_filepath("/etc/syslog-ng/coloss-analyzer-keystore.jks")
                    java_keystore_password("terces")
                    java_truststore_filepath("/etc/elasticsearch/coloss/truststore.jks")
                    java_truststore_password("terces")
                    http_auth_type("clientcert")
                    skip-cluster-health-check("yes")
                    template("$(format-json -s all-nv-pairs --rekey .SDATA.* --shift 7)")
                    time-zone("UTC")
                    type("${__es_type:-syslog}")
                  );
                };
              
Alerting

Alerting

Alerting: routing

Routing

Enriching Logs

Enriching Logs

Miscellaneous

Misc

Mailbox source - but why?

Seriously?

  • sure, for syslog-unfriendly tools
  • and quick'n'dirty solutions
  • "You Know, for Search"
  • Ex: appliances, electrical equipment, ...
  • Ex: yum, (ana)cron, ...
Mailbox source - Examples
HPSS Correlation
Correlating events using group-by()
Monitoring

Monitoring

syslog-ng-ctl
Appendices

Appendices