Each syslog-ng release comes with one or more larger features that steal the show from very useful but smaller features. Now I collect these small features and explain how they can make your life easier.
Learn what is coming up in syslog-ng version 3.18!
Larger features: HTTP batching, Python source
- Improved HTTP destination support with framing, batching and multi-threading. There are some limitations, but you can use it for example to feed Elasticsearch without needing Java.
- The new Python source allows you to create source drivers for syslog-ng in Python. There are countless log (data) sources that do not have C API, but with a ready to use Python API. The syslog-ng Python source makes it possible to integrate to these APIs.
- Template support for the Python destination adds even more flexibility.
I will cover these topics in depth in later posts. Now I focus on the many small but useful features of the 3.18 release.
Query the version of Python used by mod-python
Linux distributions are finally phasing out Python version 2 and making Python 3 the default Python implementation. Originally syslog-ng only supported version 2, but recent releases also support version 3 of Python. Once support for Python 3 in syslog-ng became available, distribution and unofficial packages are gradually switched to use Python 3. Of course you can use some package management magic to figure out which Python version syslog-ng supports, but there is now an easier, distribution neutral solution as well. The output of syslog-ng --module-registry now prints the Python version in its output:
# syslog-ng --module-registry
[...]
Module: mod-python
Status: ok
Version: 3.18.1
Core-Revision: 3.18
Description:
The python (python3) module provides Python scripted destination support for syslog-ng.
Plugins:
destination python
source python
source python_fetcher
root python
parser python
template-func python
[...]Line breaks in templates
Did you ever have to scroll your editor because syslog-ng did not support line breaks in templates? You can now break long templates into shorter lines and make your configuration easier to read. It works for template functions, for example, format-json:
file("whatever"
template("$(format-json
foo=$FOO
bar=$BAR)\n"));Or for regular text, like in this header in an http() destination:
http(headers("Authorization: \
basic abcdeffe"));Here you escape the new line with a backslash -- just like in bash, C and similar grammars -- so it does not appear in the output.
Showing the effective configuration
The new config option of syslog-ng-ctl prints the primary configuration that syslog-ng is currently using on the screen. Many people leave the main syslog-ng.conf alone, and instead of that they put local customizations into the include directory referenced in syslog-ng.conf (usually /etc/syslog-ng/conf.d). You can print the complete configuration (with included files) using the --preprocessed option:
# syslog-ng-ctl config --preprocessed
@version:3.17
@include "scl.conf"
#############################################################################
# Copyright (c) 2010-2014 Balabit
#
# This program is free software; you can redistribute it and/or modify it
[...]Note that if you include scl.conf (as in the default configuration), the output will be many pages long, as all the configuration snippets shipped with syslog-ng will also be included.
HDFS time-reap
When using the new time-reap() option for the HDFS destination, syslog-ng closes any files that has not been written to for seconds specified in this option. If archiving is configured, that is also triggered automatically for the closed file. The default value of this option is zero, which disables this timeout mechanism.
New template functions
The base64-encode template function encodes its argument using base64, as required by some APIs. If you have some incoming log messages URL encoded, then the new url-decode template function can decode it.
No more bundled mongo-c-driver source
For many practical reasons the syslog-ng source code also included bundled mongo-c-driver sources as well. Recently it caused more problem than advantages, so from now on, mongo-c-driver sources are not included in the syslog-ng source tarball. If you compile or package syslog-ng with MongoDB support, you need to make sure that development files for mongo-c-driver are available in your build environment.
Want to learn more?
If you want to learn more about the new features, changes, or bug fixes, check the syslog-ng 3.18 release notes. It also includes links to individual pull requests with source code changes and a short descriptions of these changes. For more details, see the documentation of syslog-ng 3.18.
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a list of possibilities, check the "Community" section of the syslog-ng GitHub page at https://github.com/balabit/syslog-ng. On Twitter, I am available as @PCzanik.