Network traffic is expensive in the cloud, and even a single syslog-ng instance can easily saturate the full bandwidth of a network connection. Compressing HTTP traffic was introduced in syslog-ng Version 4.4.0 and depending on your use case, you can cut down on your expenses on your networking or send more logs using the same budget or bandwidth.
Development of this feature was done using a locally installed OpenResty web server, and later tested using Sumologic. However, according to the docs it should also work with Splunk, Elasticsearch, and many other services accessible using the http() destination.
Before you begin
The possibility to compress HTTP traffic was introduced in syslog-ng Version 4.4.0. As of the time of writing this blog, most Linux distributions still carry an older version. You can use one of the unofficial repositories to install syslog-ng.
Some implementation details
The http() destination of syslog-ng is built on the curl library. It supports four different kinds of compression algorithms, out of which currently only two are available in syslog-ng: deflate and gzip. These are also the most common compression algorithms, hence why syslog-ng supports these two. Compression is available through the zlib library. Zlib is configured to use more RAM as it allows for a lesser stress on the CPU, just as using compression at level 6.
HTTP headers are not compressed, but the body is compressed. It is especially efficient when you turn on batching, as it allows to compress larger amount of logs at once. Even without using maximum compression, I often measured better than 1:5 compression on log data.
Configuring syslog-ng
The http() destination received two new compression-related options:
-
accept-encoding() is for HTTP responses. Even if syslog-ng does not process it, incoming data still contributes network traffic. Setting it to “all” ensures that syslog-ng accepts any kind of compression supported by curl.
-
Content-compression() is for log data sent by syslog-ng. Here you can choose between “deflate” and “gzip”, depending on your preference and / or what the server side supports.
A very simple example of a http() destination with compression enabled looks as follows:
destination d_http_compressed{
http(url("127.0.0.1:80"), content-compression("deflate"), accept-encoding("all"));
};
This sends log messages to the localhost using the “deflate” compression algorithm and accepts any kind of compressed response.
I mention the http() destination, while Elasticsearch, Sumologic and others have dedicated drivers, like elasticsearch-http() and so on. This is not a contradiction, these are practically wrappers around the http() destination. This also means that they accept the options of the http() destination, including compression-related options.
What is next?
This is just one of the features in the newly released syslog-ng 4.4.0. Right now, you can learn about them from the release notes at: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.4.0
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.