Actions

Tags

Zoltan.Pallagi
15 Apr 2019

This feature is available from syslog-ng PE 7.0.14 and syslog-ng OSE 3.21 on.

The configuration is really simple: - you should use the elasticsearch-http() destination (which is based on http destination).
syslog-ng will use the Elasticsearch Bulk API (https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html).

Example:

destination d_elasticsearch_http {
  elasticsearch-http(
    index("my-index")
    type("my-type")
    url("http://es-node1:9200/_bulk" "http://es-node2:9200/_bulk" "http://es-node3:9200/_bulk")
    persist-name("my-persist"));
};

The benefits are obvious: you don’t need to install and maintain any third-party dependencies (for example, Java files) like you used to earlier.
The elasticsearch-http() destination basically works with any Elasticsearch version that supports the HTTP Bulk API. Nevertheless, we tested it with Elasticsearch 6.5 and 7.0.

So let’s give it a try:
First start syslog-ng in debug mode and send a message:

root@test21:~# /opt/syslog-ng/sbin/syslog-ng -Fevdt

[2019-04-12T10:24:45.182063] cURL debug; worker='0', type='header_out', data='POST http://10.140.0.21:9200/_bulk HTTP/1.1..Host: 10.140.0.21:9200..User-Agent: syslog-ng 7.0.0+20190411+1502/libcurl 7.62.0-DEV..Accept: */*..Proxy-Connection: Keep-Alive..Content-Type: application/x-ndjson..Content-Length: 430....'
[2019-04-12T10:24:45.182086] cURL debug; worker='0', type='data_out', data='{"index":{"_type":"slng_test_type","_index":"t341f0b55f1c34b21a809_testdb"}}.{"PRIORITY":"notice","MESSAGE":"2019-04-12T10:41:15 localhost prg00000[1234]: seq: 0000000000, thread: 0000, runid: 1555058475, stamp: 2019-04-12T10:41:15 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD","ISODATE":"2019-04-12T10:24:45+02:00","HOST":"169.254.1.20","FACILITY":"user","@timestamp":"2019-04-12T10:24:45+02:00"}.'
[2019-04-12T10:24:45.182097] cURL debug; worker='0', type='text', data='upload completely sent off: 430 out of 430 bytes.'

…

[2019-04-12T10:24:45.200327] curl: HTTP response received; url='http://10.140.0.21:9200/_bulk', status_code='200', body_size='430', batch_size='1', redirected='0', total_time='0.021', worker_index='0', driver='d_elasticsearch_http#0', location='#buffer:4:3'

As you can see, the output sent to Elasticsearch is formatted as json. If syslog-ng receives the HTTP 200 return code, everything is fine.

Sending data to multiple indexes

Luckily, index() supports macros as well, so just extend it:

destination d_elasticsearch_http {
elasticsearch-http(
  index("my-index-${HOST}-${DAY}")
  type("my-type")
  url("http://es-node1:9200/_bulk"));
};

That was easy, wasn’t it?

Performance

Now go a little bit deeper. All we need is just a small performance test, using a single Elastic node.

Using the default configuration and a single index, the result on my test computer was the following:

average rate = 25385.23 msg/sec, count=762454, time=30.035, (average) msg size=392, bandwidth=9712.19 kB/sec

Using multiple indexes

Next I added the ${PROGRAM} macro to the index (my logs were generated by 8 different programs), so this way syslog-ng forwarded the logs to 8 indexes.

index("my-index") -> index("my-index-${PROGRAM}")

average rate = 74075.14 msg/sec, count=2222861, time=30.008, (average) msg size=392, bandwidth=28340.56 kB/sec

Finally based on your environment, you can fine tune the following options as well:

Increasing workers() -> by default syslog-ng uses 4 workers

Increasing batch-lines() -> by default syslog-ng uses 100 as a batch

Some tips & tricks

Since syslog-ng sends messages as json to Elasticsearch, the more complex the json the slower the speed. Json formatting is an expensive operation.

Always make sure that you configured syslog-ng to format only the required number of json message parts and not to include unnecessary macros.

In essence, avoid creating more complex json output than necessary.

Finally here is my full configuration:

@version: 7.0
@include "scl.conf"

options {
time_reopen(3);
stats_level(3);
keep_hostname(yes);
};

source s_network {
  network(
    ip("0.0.0.0")
    log_iw_size(10000)
    log_fetch_limit(1000)
    port(514));
};

destination d_elasticsearch_http {
  elasticsearch-http(
    index("t341f0b55f1c34b21a809_testdb_${PROGRAM}")
    type("slng_test_type")
    url("http://10.140.0.21:9200/_bulk")
    workers(4)
    batch_lines(100));
};

log {
source(s_network);
destination(d_elasticsearch_http);
flags(flow-control);
};
Related Content