syslog-ng 3.8 – what changed?

Almost a year has passed since the last major syslog-ng release. The first beta of the upcoming 3.8 release was published last week. This brought many changes both in terms of new features and in packaging. To encourage testing I would like to highlight some of the most important new features. Most people prefer using packages, so I also collected what changed in packaging.

Highlights

In the past few days I asked people about what they consider the most important new features in syslog-ng 3.8. Almost everybody highlighted something different, based on personal needs and preferences. The list below is in no particular order and it is far from complete, only including those that were mentioned most frequently. Some of the new features were already introduced in a blog, others are only documented in a commit message.

• Disk-based buffering can be used for storing messages on the local hard disk if the central log server or the network connection to the server becomes unavailable. Blog: https://syslog-ng.com/open-source-log-management
• Use of the official MongoDB C driver. It has many advantages and it is future-proof, but it has also brought some configuration changes.
• Elasticsearch 2.X support with Shield: https://github.com/balabit/syslog-ng/commit/cdea3e880e35751297907932dbc1aa2e917b0ec9 and https://github.com/balabit/syslog-ng/commit/8393dc1deb0ebfd74fdcbdae0d587c19f321ae4b
• Elasticsearch HTTP API support: https://github.com/balabit/syslog-ng/commit/d7f43d1aa7d1e6dc60250556c1ee1ec6cbeb7972 which also supports Elasticsearch 5
• Kafka 0.9 support
• The groupingby() parser can correlate and aggregate information independent from PatternDB. Blog: https://www.syslog-ng.com/community/b/blog/posts/the-grouping-by-parser-in-syslog-ng-3-8
• Support for Rust-based parsers has been added. The parsers are not part of the syslog-ng codebase but they are available from a separate repository: https://github.com/ihrwein/syslog-ng-rust-modules
• Added min(), max() and sum() template functions: https://github.com/balabit/syslog-ng/commit/c57720ff370951ce98fb346e0b93e4e99310d926
• Curl (HTTP) destination: https://github.com/balabit/syslog-ng/pull/978
• add-context-data syslog-ng can use an external database file to append custom name-value pairs on incoming logs: https://github.com/balabit/syslog-ng/pull/1129
• Improved the key-value parser: now it is possible to use other characters instead of the equal sign.
• Many optimizations under the hood to speed up name-value pair handling.
• Add experimental CMake build scripts: https://github.com/balabit/syslog-ng/pull/1051
• Debian packaging included: https://github.com/balabit/syslog-ng/pull/1125
• Plugin skeleton creator: https://github.com/balabit/syslog-ng/pull/1124

Packages

There are already some unofficial packages available for various Linux distributions. Most syslog-ng features are enabled in these builds, because distribution packaging guidelines are not always strictly followed by these packages  If you use an RPM-based distribution, experimental builds for some of the Rust modules are also available. In all cases it is not enough to download the syslog-ng package, you need to add the repository containing the package to be able to install all necessary dependencies.

Debian / Ubuntu: https://build.opensuse.org/package/show/home:laszlo_budai:syslog-ng/syslog-ng-3.8beta

Fedora / RHEL 7: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/

openSUSE / SLES 12: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/

Compilation / packaging changes

There are some new and changed ./configure arguments:

• –datadir now appends syslog-ng to the path name, so you need to remove it:

[codesyntax lang=”bash”]

– –datadir=”%{_datadir}/syslog-ng” \

+ –datadir=”%{_datadir}” \
[/codesyntax]

• Java modules can now be compiled independently from Java support. Therefore, even if your distribution does not have Gradle or some of the JAR dependencies to build the Java modules, you can still provide Java support in the official distribution syslog-ng package and let the users download Java modules separately. You can use the –enable-java-modules or –disable-java-modules arguments to decide whether you want to build modules.

Depending on how you package syslog-ng, you might not need to follow up all of the file list changes:

• dqtool: a tool to manage disk buffers
• new SCL configuration snippets, mostly related to Logging as a Service (LaaS) providers
• new modules, related to disk buffer, CEF formatting, and so on.
• native connector: files required to build parsers written in Rust includes a new JAR file for Elasticsearch 2.X support together with a list of JAR dependencies, which make REST API support possible

As JAR dependencies that are required for Elasticsearch 1.X and 2.X support conflict with each other, it is no more possible to store all dependencies in a flat directory. This also means that there is no single archive for building and running syslog-ng Java modules. If you need an offline version of JAR files that are required to build syslog-ng Java modules, you can use the following script to build a local Maven repository from a Gradle cache: https://github.com/lbudai/gradle_cache_to_local_maven_repo

If you have any questions or comments regarding syslog-ng 3.8 there are many ways to contact us. These are collected at https://www.syslog-ng.com/contact-sales/