Getting started with syslog-ng 3.7.1 and Elasticsearch / Hadoop / Kafka

Usually we do not publish a “Getting Started Guide” for new syslog-ng releases, as executing the “apt-get upgrade & apt-get update” commands or adding a new repo and running “yum update” is usually enough. With syslog-ng 3.7.1 it is the same, as long as you do not use Java-based destinations, like Elasticsearch, Kafka or Hadoop.

As you have probably realized by now, the major news for syslog-ng 3.7 is the addition of new Java-based destinations. Of course, there are plenty of other changes, for example Python destination, debugger, new parsers, IPv6 filtering, and several other improvements and fixes. For the complete list of changes, read the release notes at https://github.com/balabit/syslog-ng/releases/tag/syslog-ng-3.7.1.

Since syslog-ng 3.3, there are no official binaries available from BalaBit, only the source code is provided. It is available at https://github.com/balabit/syslog-ng/releases/download/syslog-ng-3.7.1/syslog-ng-3.7.1.tar.gz.

If you want to install the latest syslog-ng, you can build it on your own, wait for a distribution to update syslog-ng (but it is usually slow, as new releases are not distributed too frequently…), or use third party repositories to install syslog-ng. The repositories listed below contain syslog-ng 3.7.1 packaged for various Linux distributions. If you use any features that are only available in the syslog-ng Incubator (for example Lua support, C-based Kafka support, and so on) and download syslog-ng 3.7.1, you will lose access to these features, because syslog-ng Incubator has not yet been packaged. Hopefully syslog-ng Incubator will be packaged in the next few weeks as well.

• Debian / Ubuntu: https://build.opensuse.org/project/show/home:laszlo_budai:syslog-ng
• Fedora / EPEL7: https://copr.fedoraproject.org/coprs/czanik/syslog-ng37/
• openSUSE / SLES: https://build.opensuse.org/project/show/home:czanik:syslog-ng37 (no Java/Python for SLES11)

3.7.1 is available for FreeBSD as well. The syslog-ng 3.7.1 application is included in the sysutils/syslog-ng-devel port: http://www.freshports.org/sysutils/syslog-ng-devel/.

If you want to use the new Java-based destination drivers, life is not (yet) so easy. One difficulty is related to how Java-based drivers are built, the other one is related to how the actual Java module is built.

Compiling drivers requires a recent Gradle release, which is missing from most of the Linux distributions (the binary distribution downloaded from the Gradle website is sufficient for this purpose).

The main advantage of the Java-based destinations is, that they can use official Java based-drivers for Elasticsearch, Hadoop, and so on. But it also means that JAR files for these technologies are required both during building and using the Java-based destination drivers of syslog-ng. To make compilation and / or packaging even more difficult, right now there are no “./configure” options or environment variables. These options or variables could be used to determine the driver to be included, or the path to the JAR files, just as the rest of the destination drivers in syslog-ng use them. You have to edit Gradle configuration files directly: more about it at the end.

The current workaround (dirty hack) is to compile all drivers and provide a huge collection of JAR files that are required both during compiling and using the Java-based destination drivers of syslog-ng. The package is called syslog-ng-java-hack in the RPM directories, and it is an optional download, not a mandatory dependence. This allows you to use the JAR files from your distribution where available, or use the provided RPM for convenience.

In case of FreeBSD, all the JARs are downloaded during compilation (therefore you will need a stable network connection during package build), but there is no package that contains them. This means that you have to install Elasticsearch or Hadoop from ports to use the drivers.

The other problem is starting syslog-ng. The Java module of syslog-ng is linked to libjvm.so, that is not included in the regular library directories, so you will most likely receive the following error message on start:

Error opening plugin module; module=’mod-java’, error=’libjvm.so: cannot open shared object file: No such file or directory’

There are a couple of ways to resolve this issue:

• Add the path to libjvm.so to /etc/ld.so.config (or similar, depending on your distribution)
• Set an environment variable before starting syslog-ng, for example: LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:$LD_LIBRARY_PATH
• Add the abovementioned environment variable to a file defining the environment for the distribution-specific startup script (/etc/sysconfig/syslog-ng for RPM distributions, /etc/rc.conf for FreeBSD)

Hint: while the exact path changes with the update of Java, there are often some simplified links available, for example on Fedora-based distributions. Just compare
/usr/lib/jvm/java/jre/lib/amd64/server
with

/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.51-4.b16.fc22.x86_64/jre/lib/amd64/server/
The first (simplified) one is not only shorter, but also stays the same for the next Java update, making it unnecessary to constantly editing your configuration files.

Once the Java module and dependencies are on the machine and the library path is set, you are ready to configure syslog-ng. This part is already well-documented for Elasticsearch, Hadoop and Kafka in the destination part of the syslog-ng 3.7 documentation: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide

If you use the package with all the required JAR files, make sure that the class-path variable also includes /usr/lib/syslog-ng-java-module-dependency-jars/jars/ next to the syslog-ng Java module directory (usually: /usr/lib64/syslog-ng/java-modules). Otherwise, set it to the individual directories containing the JARs for Elasticsearch, Hadoop and / or Kafka.

If you compile syslog-ng from source, these few extra hints can be useful for you:

• openssl is now a mandatory dependence (until 3.6 is was highly recommended, but optional).
• json-c is highly recommended, but still optional (it might cause some trouble, if you use the system() source, update from 3.6 and forget to update the version to 3.7 in the configuration file).
• If you want man pages, you need to enable it explicitly in ./configure using the –enable-manpages option.
• You can influence which Java destination drivers are compiled by editing modules/java-modules/settings.gradle
• You can set where Gradle looks for JAR files on the local file system by editing modules/java-modules/build.gradle