syslog-ng with Elastic Stack 7

For many years, anything I wrote about syslog-ng and Elasticsearch was valid for all available versions. Well, not anymore. With version 7 of Elasticsearch, there are some breaking changes. These changes are mostly related to the fact that Elastic is phasing out type support. This effects mapping (as the _default_ keyword is not used any more) and the syslog-ng configuration as well (as even though type() is a mandatory parameter, you should leave it empty).

This blog post is a rewrite of one of my earlier blog posts (about creating a heat map using syslog-ng + Elasticsearch + Kibana), focusing on the changes and the new elasticsearch-http() destination:

https://www.syslog-ng.com/community/b/blog/posts/creating-heat-maps-using-new-syslog-ng-geoip2-parser

Before you begin

First of all, you need some iptables log messages. In my case, I used logs from my Turris Omnia router. If you do not have iptables logs at hand, there are many sample logs available on the Internet. For example, you can use Bundle 2 from Anton Chuvakin( http://log-sharing.dreamhosters.com/ ) and use loggen to feed it to syslog-ng.

Then, you also need a recent version of syslog-ng. The elasticsearch-http() destination was introduced in syslog-ng (OSE) version 3.21.1 (and PE version 7.0.14).

Last but not least, you will also need Elasticsearch and Kibana installed. I used version 7.1.1 of the Elastic Stack, but any version after 7.0.0 should be fine. Using 7.1.0 or later has the added benefit of having basic security built in for free: neither payment nor installing extensions arenecessary.

Mapping

Mapping in Elasticsearch defines how a document and the fields it contains are stored and indexed. You do not have to configure mapping by hand for generic syslog data, but you need it for storing geolocations, for example. Starting with Elasticsearch version 7, the use of the _default_ keyword is not supported, so when you try to apply the mapping example from my previous blog post, it will fail with an error message. To avoid this, apply the following to the index called syslog-ng before starting to send log messages:

{
   "mappings" : {
         "properties" : {
            "geoip2" : {
               "properties" : {
                  "location2" : {
                     "type" : "geo_point"
                  }
               }
            }
         }
   }
}

Configuring syslog-ng

The configuration below is slightly different from the one in my original blog post. Here I would like to emphasize only the differences:

source s_tcp {
  tcp(ip("0.0.0.0") port("514"));
};
 
parser p_kv {kv-parser(prefix("kv.")); };
 
parser p_geoip2 { geoip2( "${kv.SRC}", prefix( "geoip2." ) database( "/usr/share/GeoIP/GeoLite2-City.mmdb" ) ); };
 
rewrite r_geoip2 {
    set(
        "${geoip2.location.latitude},${geoip2.location.longitude}",
        value( "geoip2.location2" ),
        condition(not "${geoip2.location.latitude}" == "")
    );
};
 
destination d_elasticsearch_http {
    elasticsearch-http(
        index("syslog-ng")
        type("")
        url("http://localhost:9200/_bulk")
        template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE --key ISODATE @timestamp=${ISODATE})")
    );
};


log {
    source(s_sys);
    source(s_tcp);
    if (match("s_tcp" value("SOURCE"))) {
        parser(p_kv);
        parser(p_geoip2);
        rewrite(r_geoip2);
    };
    destination(d_elasticsearch_http);
    flags(flow-control);
};

The major differences are:

• As the log statement also includes the local logs, I added an extra filter. This way only iptables logs arriving through the network source are parsed and enriched with geolocation information.

• Instead of the Java-based Elasticsearch destination, we use the new elasticsearch-http() destination (based on the http() destination written in C).

• Normally, type() is a mandatory parameter, but for Elasticsearch 7 you can leave it empty. You can do that using quotation marks: type("").

• As syslog-ng parses many logs by default (Application Adapters), dot-nv-pairs are added to the scope. Shifting is needed as Elasticsearch does not like the fields starting with an underscore (syslog-ng creates name-value pairs starting with a dot, which are turned into underscores when transformed to JSON).

Note: on the one hand, we have received many positive feedback on the new elasticsearch-http() destination. As it does not need Java any more, it is easier to install and configure. On the other hand, we also have reports of increased network traffic (your mileage may vary).

You can read more about the changes on the Elasticsearch side (including a roadmap of changes) here: https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/balabit/syslog-ng. On Twitter, I am available as @PCzanik.