Test the latest FreeBSD with syslog-ng 3.8.1

Even if I do not maintain a cluster of FreeBSD servers with thousands of jails anymore, my favorite server operating system is still FreeBSD. I started using it at version 2.X well over twenty years ago. Yesterday, FreeBSD 11.0 was released. For a complete list of changes you can read the release notes. I will list some of my favorite ones:

  • native ZFS support in installer
  • security hardening as part of the installation
  • many jail and virtualization-related improvements
  • lot of work on ARM architecture support, including ARMv8 (64bit ARM)
  • and countless driver updates, as usual

As the FreeBSD FTP server is quite overloaded after a release, I recommend using a mirror site for downloading the installers. You can find the list of mirrors here.


Compiling syslog-ng 3.8.1 with Java module support

Java destinations on FreeBSD are not supported by the syslog-ng team and for technical reasons (it downloads JAR files from the Internet) it is also marked broken in FreeBSD ports. Luckily it does not prevent it from working: some people even use it it production.

If you do not want to wait for ages for dependent packages to compile, you can have a head start by using pkg. The following commands will install the latest OpenJDK, Gradle and syslog-ng together with their dependencies. Depending on the machine you use for compiling syslog-ng, this can give you an hour or half a day advantage. At the end you can delete syslog-ng-devel, as you compile it with new configuration options.

Enter the following commands:


pkg install syslog-ng-devel
 
pkg install gradle
 
pkg install openjdk8
 
pkg remove syslog-ng-devel

Make sure that you have the latest available ports tree:

portsnap fetch && portsnap extract

You can also use the “update” optiom instead of “extract” if you have already used Portsnap on that machine previously. Once you have an up-to-date ports tree, change to the /usr/ports/sysutils/syslog-ng-devel directory, which contains syslog-ng 3.8.1 by the time of writing (I tend to point the sysutils/syslog-ng port to the latest stable release only once a .2 version is available). Open the Makefile in your favorite text editor, and make sure that the following line is deleted or commented out:

“JAVA_MOD_BROKEN= Unsupported maven downloads.”

You can now configure the port using “make config”. Make sure, that both “Java” and “Java modules” are enabled. Once it is configured, you can build and install syslog-ng with Java-based destination drivers. Note that downloading JAR dependencies requires an active Internet connection without a proxy.


Configuring a Java destination on FreeBSD

Java destination drivers in syslog-ng require access to libjvm.so to be able to run. This is not available by default, as you might need multiple Java versions on your machine.

If you are sure that only one version is in use, the following commands make it available for syslog-ng:


echo /usr/local/openjdk8/jre/lib/amd64/server > /usr/local/libdata/ldconfig/java
 
/etc/rc.d/ldconfig restart

Another method is to add the following path to the LD_LIBRARY_PATH environment variable on the command line or in the script starting syslog-ng:

/usr/local/openjdk8/jre/lib/amd64/server

When you compiled the Java modules, some JAR files were downloaded from the Internet. Unfortunately there is no easy way to reuse them for run-time, therefore you have to follow the documentation and download JAR files accordingly.

If you want to use Elasticsearch, there is an easier way doing that: just install Elasticsearch from ports. If you use pkg, enter the following command:

pkg install elasticsearch2

After this, configure client_lib_dir to point to the freshly installed JAR files. Check the documentation for a detailed explanation of configuration options. To do this, add the following lines to the syslog-ng configuration file:


destination d_elasticsearch23 {
 
elasticsearch2(
 
client_lib_dir(/usr/local/lib/elasticsearch/lib)
 
index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
 
type("test")
 
cluster("elasticsearch")
 
flush_limit("1000")
 
template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")
 
);
 
};

If you are replacing Logstash and want to get started with parsing messages with syslog-ng – replacing grok parsers – see the following links:

For practical details about how to get started storing logs in Elasticsearch using syslog-ng, see our Elasticsearch white paper.


Are you stuck?

If you have any questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. 

Related Content