Jump-starting ESK: Elasticsearch, syslog-ng and Kibana

If you want to test drive syslog-ng or just want to learn something new, I recommend you checking out the BLACK ESK project. By running a single script, you can set up a containerized test environment, complete with Elasticsearch, Kibana and a syslog-ng server. All network connections among them are encrypted and the syslog-ng configuration showcases many interesting syslog-ng features, including PatternDB and JSON parsing, GeoIP, in-list filtering and the new Elasticsearch destination. Once it is installed, all you need are some logs directed at this server and a browser to reach Kibana. You can learn a lot from reading through the setup scripts and the different configuration files.

I must thank @uidz3ro, who created BLACK ESK and fixed a number of problems while I tested it and even implemented a little feature request. This way it is possible to create maps (like the one below mapping IP addresses from failed ssh logins) without any further configuration.

Before you begin

Getting started with BLACK ESK is easy, as it does not have much external dependencies. On the container side it needs Docker and docker-compose (I did not check Podman, as podman-compose is still under development) and you need git to check out sources from GitHub. And of course you need syslog to send log messages to this containerized ESK stack. In my example, I use syslog-ng, but any other should work just fine.

If you have a firewall on the host you want to install BLACK ESK, make sure that port 514 (syslog) and 5601 (Kibana) are open. The syslog-ng configuration file lists a few more ports, but those are not forwarded by the supplied Docker configuration.

Installation

As a first step, you have to check it out from GitHub:

git clone https://github.com/amitn322/blackesk

Change to the freshly created blackesk directory and start the installer script:

cd blackesk
sh install.sh single-node

You can now sit back and relax. It takes some time to download and build all the necessary software. The “single-node” parameter here means that only a single Elasticsearch container is created. The installation script prints many lines on your screen. At the end, you should see something like this on your terminal:

Generating Some Fake Logs, you can delete the index and start over..
10 Logs generated
Everything should have been completed, please login to https://ipaddress:5601 with following user:
--------------------------------------------------
Username: elastic
Password: pQc85zA7vAnRIL5JaKL0
--------------------------------------------------
The initial set of credentials are stored in .creds.txt in the current Directory

There are many things to note here:

• There are some fake logs uploaded by the installer, so you can easily check how the web interface works even before you send your own data

• As all passwords are randomly generated, I have no problem sharing the actual output here

• Make sure that you enter “https://” before the IP address, as unencrypted communication does not work here and browsers give all sorts of strange error messages (know by experience…).

• Replace “ipaddress” with the IP address of the host, where you installed BLACK ESK

• Do not worry if the user name and password are not retained on your terminal: they are also saved in the .creds.txt file

You can now log in and look around in Kibana. Click on “Discover” and you should see some log messages from syslog-ng.

Starting/stopping

For me, this was the first time I used docker-compose, so it took a while to figure out how to stop and restart the containers. Assuming that you successfully installed BLACK ESK and after some experimenting you want to stop it, enter the following command while in the blackesk directory:

docker-compose -f docker-compose-singlenode.yml stop

And you can start it up again using:

docker-compose -f docker-compose-singlenode.yml up -d

Getting logs

Obviously, it is not much worth having syslog-ng with a NoSQL back end and a fancy graphical user interface without logs. The bundled configuration expects legacy (RFC 3164) syslog messages on port 514 using both TCP and UDP. The easiest is to install syslog-ng on the host running BLACK ESK and forward all logs to port 514. Just add some similar lines to your syslog-ng configuration:

destination d_blackesk {
    tcp("127.0.0.1" port("514"));
};
log {source(src); destination(d_blackesk);};

The name of the source might be different on your system, “src” is used in the syslog-ng.conf of openSUSE/SLES, “s_sys” on Fedora/RHEL/CentOS.

A similar configuration snippet but with a different IP address could be used from other hosts on your network.

Playing with syslog-ng

BLACK ESK comes with a nice and complex syslog-ng configuration showcasing many syslog-ng features. It includes JSON and PatternDB message parsing, many different kinds of filters, including the in-list filter for black- and whitelisting, GeoIP to find the geolocation of IP addresses and of course an Elasticsearch destination.

Luckily the syslog-ng configuration is available in an easy to access location right in your blackesk directory, under syslog-ng/conf/syslog-ng.conf. Before making any changes, create a backup of this file. Then open it in your favorite text editor and start making changes. Do not forget to check your changes before reloading syslog-ng:

localhost:~/blackesk # docker exec syslog01 syslog-ng -s
Error parsing afsocket, inner-src plugin myerror not found in /etc/syslog-ng/syslog-ng.conf:40:22-40:29:
35      source s_windows{
36      	tcp( ip(0.0.0.0) port(5515) flags(no-parse));
37      };
38      
39      source s_network {
40---->     tcp(ip(0.0.0.0), myerror(), port(514));
40---->                      ^^^^^^^
41          udp(ip(0.0.0.0), port(514));
42          unix-stream("/var/run/syslog-ng/syslog-ng.sock");
43      };
44      
45      #Templates

This runs a syntax check in an already running syslog-ng container. Here I added an error just to demo it. Once the syntax check runs without error, you can make your configuration live:

localhost:~/blackesk # docker exec syslog01 syslog-ng-ctl reload
Config reload successful

Of course, some spelling mistakes are not spotted by this, but can still ruin your configuration. In this case try to fix your configuration and then start the container using:

docker start syslog01

Uninstalling

There is also an uninstall script included. Just enter:

sh uninstall.sh

Note, that it removes all data, but not the container images. You have to do that manually. It is also an easy way to hard reset your test environment and do a fresh start using install.sh. In this case the install script runs much faster, as it does not need to create the container images, just the keys for TSL and a few configurations.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.