Version 7 of the Elastic Stack, packed with new features and improved performance, has now been available for some time. Elasticsearch is not the only one to have come up with a major new version recently: starting with version 3.21, syslog-ng features a new Elasticsearch destination driver (based on the http() destination) that does not require Java. In most cases, it is more resource friendly than the Java-based driver and it is definitely easier to configure. It also has the benefit that unlike the Java-based driver, it can be included in Linux distributions.
This is a quick how-to guide to get you started with syslog-ng (the current version being 3.21) and Elasticsearch 7 on RHEL/CentOS 7.
If you use GeoIP in syslog-ng and mapping in Kibana, I also recommend reading my blog post on related changes at https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-with-elastic-stack-7
Installing applications
As a first step, you have to enable a number of software repositories, and then install applications from them. (These repositories contain Elasticsearch, the latest version of syslog-ng, and the dependencies of syslog-ng, and are all required for Elasticsearch 7 support.)
-
Using RHEL: You first have to enable the so-called “optional” repository (or repo, in its more popular shorter form), which contains a number of packages required to start syslog-ng.
Using CentOS: The contents of the “optional” repo are included in CentOS, so you do not have to enable the repo separately there.
subscription-manager repos --enable rhel-7-server-optional-rpms
-
The Extra Packages for Enterprise Linux (EPEL) contain many useful packages not included in RHEL. EPEL also has an older version of syslog-ng, but that version does not support Elasticsearch at all. Still, a few dependencies of syslog-ng come from this repo, which you can enable by downloading and installing an RPM package:
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm rpm -Uvh epel-release-latest-7.noarch.rpm
-
Next, add the repo containing the latest unofficial build of syslog-ng. At the time of this blog post, it is syslog-ng 3.13 (available on the Copr Build Service). Download the repo file to /etc/yum.repos.d/, so you can install and enable syslog-ng:
cd /etc/yum.repos.d/ wget https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng321/repo/epel-7/czanik-syslog-ng321-epel-7.repo yum install syslog-ng yum install syslog-ng-http systemctl enable syslog-ng systemctl start syslog-ng
-
While it is not strictly required, you can avoid some confusion if you also delete rsyslog at the same time:
yum erase rsyslog
-
To install Elasticsearch, you have to use your text editing skills: copy and paste repository information from https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html into a file under /etc/yum.repos.d:
cd /etc/yum.repos.d/ vi elastic.repo yum install elasticsearch
Configuring syslog-ng
As a last step, create a configuration file for syslog-ng. A base configuration is already in place. You can extend it by creating a file under /etc/syslog-ng/conf.d with a .conf extension.
cd /etc/syslog-ng/conf.d vi es.conf
The following configuration has a few twists, making it possible to have some name-value pairs to analyze without the need to write PatternDB rules.The complete configuration will be included at the end of this section; the configuration snippets are used to demonstrate the role of each part.
The first part of the configuration defines a file source for audit.log.
source s_auditd {
file(/var/log/audit/audit.log);
};
The next part defines the Elasticsearch destination. Here we use the new elasticsearch-http() destination instead of the old Java-based elasticsearch() destination. For backwards compatibility, type() is a mandatory parameter on the syslog-ng side.For Elasticsearch 7, you should leave it empty. You can do that using quotation marks: type("").
destination d_elastic {
elasticsearch-http(
index("syslog-ng")
type("")
url("http://localhost:9200/_bulk")
template("$(format-json --scope rfc5424
--scope nv-pairs --exclude DATE --key ISODATE)")
);
};
The first log path sends local logs to the Elasticsearch destination without any processing. The source of the local logs, source(s_sys), is defined in /etc/syslog-ng/syslog-ng.conf, the main configuration file of syslog-ng.
log {
source(s_sys);
destination(d_elastic);
};
The second log path parses audit.log with the Linux audit parser, and further parses the MSG field of audit logs, which can contain valuable information (for example, source IP address and the status of an SSH login). Just like the previous log path, this one also stores the results to Elasticsearch, but in this case, it includes many interesting name-value pairs.
log {
source(s_auditd);
parser {
linux-audit-parser (prefix("auditd."));
};
parser {
kv-parser (template("${auditd.msg}") prefix("amsg."));
};
destination(d_elastic);
};
Finally, here is the complete configuration to make copy & paste easier for you:
source s_auditd {
file(/var/log/audit/audit.log);
};
destination d_elastic {
elasticsearch-http(
index("syslog-ng")
type("")
url("http://localhost:9200/_bulk")
template("$(format-json --scope rfc5424
--scope nv-pairs --exclude DATE --key ISODATE)")
);
};
log {
source(s_sys);
destination(d_elastic);
};
log {
source(s_auditd);
parser {
linux-audit-parser (prefix("auditd."));
};
parser {
kv-parser (template("${auditd.msg}") prefix("amsg."));
};
destination(d_elastic);
};
Displaying results
Most people use Elasticsearch because they want to use Kibana to search and visualize their log messages. To set up Kibana:
-
Install Kibana using the previously configured Elastic repository and issuing the following command:
yum install kibana
-
By default, the Kibana web interface binds to 127.0.0.1 only, making the inferface inaccessible if you want to view it from a remote machine. If you want to reach Kibana remotely, change the server.host setting in /etc/kibana/kibana.yml to 0.0.0.0 or the server's IP address.
-
You can now enable and start Kibana:
systemctl enable kibana systemctl start kibana
-
When you first open Kibana on port 5601, it will display an initial setup screen. You have to enter the “syslog-ng*” index name here – that is, if you have followed my instructions above and used the same index name.
-
Once Kibana has found the index, you have to configure the “Time-field name”. If you use the configuration above for syslog-ng, it is “ISODATE”.
-
Click Create, and Kibana is ready to use.
Are you stuck?
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a long list of possibilities, check our contact page at https://syslog-ng.org/contact-us/. On Twitter I am available as @PCzanik.