For the last six months, Elastic’s communication centered around the upcoming Elastic Stack 5.0. And finally it is here: tons of new features, improved performance and a single version number for all Elastic products. Compatibility with syslog-ng was checked already during the alpha phase of development, as syslog-ng is becoming popular among Elasticsearch users: it can greatly simplify logging to Elasticsearch.

As Elastic Stack 5.0.0 is now generally available, here is a quick how-to guide to get you started with syslog-ng 3.8.1 and Elasticsearch 5.0.0 on RHEL/CentOS 7.

This is the fourth blog post in a six-part series on storing logs in Elasticsearch using syslog-ng. You’ll find a link to the next and previous parts in the series at the end of this post. You can also read the whole Elasticsearch series in a single white paper.

Installing applications

As a first step, you have to enable a number of software repositories, and then install applications from them. These repositories contain Elasticsearch, the latest version of syslog-ng, and the dependencies of syslog-ng. These are all required for Elasticsearch 5.0.0 support.

In case of RHEL: You first have to enable the so-called “optional” repository (or repo, in its more popular shorter form), which contains a number of packages that are required to start syslog-ng.

In case of CentOS: The content of this repo is included CentOS, so you do not have to enable it there separately:

subscription-manager repos --enable rhel-7-server-optional-rpms

The Extra Packages for Enterprise Linux (EPEL) contains many useful packages, which are not included in RHEL. It also has an older version of syslog-ng, but that does not support Elasticsearch at all. Still, a few dependencies of syslog-ng are coming from this repo. You can enable it by downloading and installing an RPM package:

wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -Uvh epel-release-latest-7.noarch.rpm

Next add the repo containing the latest unofficial build of syslog-ng. By the time of writing it is syslog-ng 3.8 and it is available on the Copr build service. Download the repo file to /etc/yum.repos.d/, so you can install and enable syslog-ng:

cd /etc/yum.repos.d/
wget https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/repo/epel-7/czanik-syslog-ng38-epel-7.repoyum install syslog-ng
yum install syslog-ng-java
systemctl enable syslog-ng
systemctl start syslog-ng

It is not strictly required, but you can avoid some confusion, if you also delete rsyslog at the same time:

yum erase rsyslog

To install Elasticsearch, you have to use your text editing skills: copy and paste repository information from https://www.elastic.co/guide/en/elasticsearch/reference/5.0/rpm.html into a file under /etc/yum.repos.d

cd /etc/yum.repos.d/
vi elasticsearch.repo
yum install elasticsearch

Before starting Elasticsearch, you should change at least one setting in the configuration file: the name of the Elasticsearch cluster. Make sure there is no other cluster with the same name on your network. What you define here is to be used later also in your syslog-ng configuration. Once you have configured it, you can also enable and start Elasticsearch.

echo cluster.name: syslog-ng >> /etc/elasticsearch/elasticsearch.yml
systemctl enable elasticsearch
systemctl start elasticsearch

Java-based destinations in syslog-ng require libjvm.so in the library path.

If you only have a single Java version on your system, the commands below add the directory containing libjvm.so to the library path:

echo /usr/lib/jvm/jre/lib/amd64/server > /etc/ld.so.conf.d/java.conf
ldconfig

You can check whether syslog-ng finds the libjvm.so file using the following command:

syslog-ng -V

The version information also includes a warning message if syslog-ng can not find libjvm.so. In this case refer to the blog mentioned above to resolve the problem.

Configuring syslog-ng

As a last step, create a configuration file for syslog-ng. A base configuration is already in place. You can extend it by creating a file under /etc/syslog-ng/conf.d with a .conf extension.

cd /etc/syslog-ng/conf.d
vi es.conf

The following configuration has a few twists, making it possible to have a few name-value pairs to analyze without the need to write PatternDB rules.

The complete configuration will be included at the end of this section; the configuration snippets are used to demonstrate the role of each part.

The first part of the configuration defines a file source for audit.log.

source s_auditd {
  file(/var/log/audit/audit.log);
};

The next part defines the Elasticsearch destination. The name of the Elasticsearch cluster is “syslog-ng”. If you have configured something else as the name of the Elasticsearch cluster, use that name here. Note that the client mode must be “http”, other modes are not supported for Elasticsearch 5.0

destination d_elastic {
  elasticsearch2 (
    cluster("syslog-ng")
    client_mode("http")
    index("syslog-ng")
    type("test")
    template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")
  )
};

The first log path sends local logs to the Elasticsearch destination without any processing. The source of the local logs, source(s_sys) is defined in /etc/syslog-ng/syslog-ng.conf, the main configuration file of syslog-ng.

log {
  source(s_sys);
  destination(d_elastic);
};

The second log path parses audit.log with the Linux audit parser, and further parses the MSG field of audit logs, which can contain valuable information (for example source IP address and the status of an SSH login). Just like the other log path, this one also stores the results to Elasticsearch, but in this case it includes many interesting name-value pairs.

log {
  source(s_auditd);
  parser {
    linux-audit-parser (prefix("auditd."));
  };
  parser {
    kv-parser (template("${auditd.msg}") prefix("amsg."));
  };
  destination(d_elastic);
};

And the whole configuration ready for copy & paste:

source s_auditd {
  file(/var/log/audit/audit.log);
};
destination d_elastic {
  elasticsearch2 (
    cluster("syslog-ng")
    client_mode("http")
    index("syslog-ng")
    type("test")
    template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")
  )
};
log {
  source(s_sys);
  destination(d_elastic);
};
log {
  source(s_auditd);
  parser {
    linux-audit-parser (prefix("auditd."));
  };
  parser {
    kv-parser (template("${auditd.msg}") prefix("amsg."));
  };
  destination(d_elastic);
};

Displaying results

Most people use Elasticsearch because they want to use Kibana to search and visualize their log messages. To install Kibana, copy-paste the repo information from https://www.elastic.co/guide/en/kibana/5.0/rpm.html to a file under /etc/yum.repos.d/ and then install it using the following command:

yum install kibana

By default the Kibana web interface binds only to 127.0.0.1, making it inaccessible if you want to view it from a remote machine. Change the server.host setting in /etc/kibana/kibana.yml to the server’s IP address or to 0.0.0.0 if you want to reach Kibana remotely. You can now enable and start Kibana:

systemctl enable kibana
systemctl start kibana

When you first open Kibana on port 5601 it will display an initial setup screen. You have to enter the “syslog-ng*” index name here, if you have followed my instructions. Once Kibana has found the index, you have to configure the “Time-field name”. If you use the above configuration for syslog-ng, it is “ISODATE”. Once you click Create, Kibana is ready to use.

In my next Elasticsearch blog, I cover how to send netdata metrics through syslog-ng to Elasticsearch. I also have a blog talking about how to get started on RHEL/CentOS using syslog-ng and Elasticsearch version 6.

In my previous blogs in the Elasticsearch series, I covered:

• Basic information about using Elasticsearch with syslog-ng and how syslog-ng can simplify your logging architecture
• Logging to Elasticsearch made simple with syslog-ng
• How to parse data with syslog-ng, store in Elasticsearch, and analyze with Kibana

Read the entire series about storing logs in Elasticsearch using syslog-ng in a single white paper.