Actions

Tags

Peter Czanik
16 Feb 2017
Securing your ESK stack for free using Search Guard

This is a guest post by Fabien Wernli (faxm0dem). He has been administering Linux clusters at the Computing Centre of the National Institute of Nuclear Physics and Particle Physics (CC-IN2P3) for 10+ years. Among others, he is an expert on performance-data monitoring and infrastructure management.

Introduction

Using syslog-ng to index events to your Elasticsearch (ES) cluster has been possible natively for some time now (see various blog posts), but now with syslog-ng 3.9, you can even use Search Guard to encrypt and authenticate your connections to Elasticsearch.

The main reasons we’re using syslog-ng over Logstash (LS) at the Computing Centre of the National Institute of Nuclear Physics and Particle Physics (CC-IN2P3) are:

  • superior performance,
  • lower system footprint, and
  • higher maintainability.

One of the reasons Elasticsearch is a popular choice to store and query logs is the fact that it’s an opensource software (OSS) solution. Unfortunately, one of its drawbacks is security: the only official way to use flexible authorization policies and strong cryptography is to use Shield (recently renamed to X-pack). And guess what, that’s where the free part stops: Shield requires a full ES license, which can quickly empty your pockets, especially if your cluster grows.

Fortunately, the Search Guard (SG) project implements strong security in the form of an ES plugin. Its core features are opensource, with additional functionality requiring a license. However, even the latter code is publicly available on GitHub.

That’s where the subject of this post comes in: Search Guard support got merged to syslog-ng! Moreover, the Search Guard component required by syslog-ng doesn’t require a commercial license, so you can use it for free. Hopefully after reading this post, you’ll be able to fly your ESK stack to higher security standards.

Notes

This guide targets ES 2.4.1 (official distribution from elastic.co), and SG-ssl 2.4.1.16. Instructions should be pretty much the same for other 2.X Elasticsearch releases. If you use Search Guard with Elasticsearch 5 or later, you can use the HTTPS mode of syslog-ng to connect securely to your cluster.

Search Guard

Search Guard started as a free plugin for ES 1.x, then was rewritten completely for ES 2.x, and recently gained support for ES 5.x. Its main components are Search Guard and Search Guard SSL (both got merged since 5.x to ease deployment). We’re not going into much detail about the former, as you’ll be able to find a lot of info on its homepage and documentation. Suffice it to say that its function is authentication and authorization, and that it supports many mechanisms like http simple, proxy, kerberos, etc. Also note that its development team is very responsive to changes in ES APIs. We’ll discuss some more on the latter: Search Guard SSL is the plugin that handles encryption: it’s the only one needed by syslog-ng, because it will talk natively with the other ES nodes on the encrypted transport layer (aka the 9300 tcp port).

Search Guard SSL

Citing the project page:

Search Guard (®) SSL is a free and open source security plugin for Elasticsearch which provides SSL/TLS for the transport- and REST-layer.

Features

  • Node-to-node encryption through SSL/TLS (Transport layer)
  • Secure REST layer through HTTPS (SSL/TLS)
  • Supports JDK SSL and Open SSL
  • Works with Kibana 4, logstash and beats

We kindly invite the developers to add syslog-ng to that list as soon as this blog post is published ;-).

Configuration

In order to connect syslog-ng to your Search Guard SSL enabled ES cluster, you will need the following:

  • syslog-ng-3.9.1
  • a working search-guard-ssl enabled ES cluster
  • Elasticsearch installed on the syslog-ng node. (You need the same Elasticsearch version that the cluster uses, but it is not necessary to actually run it)
  • a copy of this blog post (or two)

For the impatient, the steps to follow are:

  1. Install the Search Guard SSL plugin on the syslog-ng node
  2. Generate a certificate for the syslog-ng node and add it to the cluster nodes’ keystore
  3. Configure syslog-ng’s elasticsearch2 destination to use client-mode(searchguard)
  4. Test
  5. Enjoy

Install search-guard plugin

sudo /usr/share/elasticsearch/bin/plugin install -b com.floragunn/search-guard-ssl/2.4.1.16

Certificates

Managing SG certificates is beyond the scope of this article. Please refer to the SG-SSL documentation regarding certificates for more details. The logical steps are the following:

  1. Generate a new key for your syslog-ng node
  2. Generate a CSR and send it to your CA (or self)
  3. CA sends back signed certificate
  4. Add the latter to the keystore of the node (SYSLOG_NG-NODE_NAME-keystore.jks)

For a quick’n’dirty PoC you can use floragunn’s certificate generator to get started in a breeze.

syslog-ng configuration

The configuration is pretty straightforward, just follow the destination’s documentation using the correct client-mode.

In the examples, 192.168.42.42 is the syslog-server, and 192.168.42.69 one of the Elasticsearch cluster nodes.

The following example will push messages to a daily index with a fixed type:

File: /etc/syslog-ng/syslog-ng.conf

destination d_elasticsearch {   elasticsearch2(   client-lib-dir("/usr/share/elasticsearch/plugins/search-guard-ssl/*.jar:/usr/share/elasticsearch/lib")   index("syslog-${YEAR}.${MONTH}.${DAY}")   type("syslog")   time-zone("UTC")   client_mode("searchguard")   resource("/etc/syslog-ng/elasticsearch.yml")   );   };

File: /etc/syslog-ng/elasticsearch.yml

cluster:   name: elasticsearch   discovery:   zen:   ping:   unicast:   hosts:   - 192.168.42.42   - 192.168.42.69   node:   name: syslog_ng_secure   data; false   master: false   path:   home: /etc/syslog-ng   conf: /etc/syslog-ng   searchguard:   ssl:   transport:   keystore_filepath: syslog_ng-keystore.jks   keystore_password: changeit   truststore_filepath: truststore.jks   truststore_password: changeit   enforce_hostname_verification: true

The key- and trust-stores should be placed into the directory path.conf = /etc/syslog-ng.

Note

The thorough reader might have frowned at the value of client-lib-dir(). We invite them to read the contents of github issue/1274 for more information on that matter. The unthorough reader may silently ignore this note.

 

Testing

The first thing to do is to test the syntax of our syslog-ng config using syslog-ng -f /etc/syslog-ng/syslog-ng.conf -s. The easiest way to test the config is then to run syslog-ng in the foreground using the command-line switch -F, and to activate the Full Monty debugging using the switches -d and -v: syslog-ng -f /etc/syslog-ng/syslog-ng.conf -Fdv.

If everything goes as expected, you should see something like the following:

[2016-12-20T13:19:08.289217] Module loaded and initialized successfully; module='mod-java'   [2016-12-20T13:19:08.303180] Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='mod-java.so', module='mod-java'   [2016-12-20T13:19:08.303201] Registering candidate plugin; module='mod-java', context='destination', name='java', preference='0'   [2016-12-20T13:19:08.452898] Add path to classpath: /usr/local/lib/syslog-ng/java-modules/java-modules.jar;   [2016-12-20T13:19:08.453308] Add path to classpath: /usr/local/lib/syslog-ng/java-modules/syslog-ng-core.jar;   [2016-12-20T13:19:08.453466] Add path to classpath: /usr/local/lib/syslog-ng/java-modules/elastic-v2.jar;   [2016-12-20T13:19:08.455703] Add path to classpath: /usr/share/elasticsearch/plugins/search-guard-ssl/search-guard-ssl-2.4.1.16.jar;   [2016-12-20T13:19:08.738063] [syslog_ng] modules [], plugins [search-guard-ssl], sites [];   […]   [2016-12-20T13:19:09.997536] Worker thread started; driver='d_elasticsearch#0'   [2016-12-20T13:19:09.997870] Checking cluster state..., cluster_name='elasticsearch';   [2016-12-20T13:19:10.043771] [id: 0xb6975e55, /192.168.42.42:55330 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;   [2016-12-20T13:19:10.052281] Cluster is ready to work, cluster_name='elasticsearch';   […now that everything's set up logs start to come in…]   [2016-12-20T13:20:34.548044] Incoming log entry; line='<27>Dec 20 13:20:34 root: /etc/dhcp/dhclient-enter-hooks.d/avahi-autoipd returned non-zero exit status 1'   […and get written to elasticsearch…]   [2016-12-20T13:20:34.550464] Outgoing message; message='2016-12-20T13:20:34+01:00 root err /etc/dhcp/dhclient-enter-hooks.d/avahi-autoipd returned non-zero exit status 1\x0a'   [2016-12-20T13:20:34.550878] Outgoing log entry, json='{"PROGRAM":"root","PRIORITY":"err","PID":"11636","MESSAGE":"/etc/dhcp/dhclient-enter-hooks.d/avahi-autoipd returned non-zero exit status 1","ISODATE":"2016-12-20T12:20:34+00:00","HOST":"ccswissrp","FACILITY":"daemon","@timestamp":"2016-12-20T12:20:34+00:00"}';   [2016-12-20T13:20:35.078198] Start bulk processing, id='1';   [2016-12-20T13:20:35.086627] using decoder[VanillaChunkDecoder] ;   [2016-12-20T13:20:35.118315] Bulk processing finished successfully, id='1', numberOfMessages='1';   […]

Conclusion

We showed it’s possible to setup a secure line between syslog-ng and Elasticsearch using Search Guard. All communication, be it between cluster nodes or between syslog-ng and the latter is handled using SSL.

Related Content