Do you have to forward large amounts of logs between two syslog-ng instances? OTLP (OpenTelemetry protocol) support in syslog-ng was contributed by Axoflow, and it can solve this problem. Just like the ewmm() destination, syslog-ng-otlp() forwards most name-value pairs, however, unlike a tcp() connection, it scales well with multiple CPU cores.
Support for OpenTelemetry was added to syslog-ng a couple of releases ago. OpenTelemetry is an observability framework, mainly used in Linux / Cloud / Kubernetes environments. However, I already had users asking to make this feature available on FreeBSD. (It already worked once, but now it fails to compile again.)
Version 4.6.0 added many new OTLP-related enhancements. Batching and multiple workers make OTLP connections significantly faster, while compression can save you bandwidth at the expense of some more CPU usage. This changes the syslog-ng-otlp() destination from an interesting experiment into something really useful. It enables you to send a lot more log messages between two syslog-ng instances than with a tcp() connection, while using less bandwidth.
Before you begin
You need syslog-ng 4.6.0 or later to test features mentioned in this blog. Right now, it is available in the various unofficial syslog-ng package repositories for Debian, Ubuntu, Fedora, openSUSE Tumbleweed and RHEL 9. You can find more information about these repositories at https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx Personally, I used openSUSE Tumbleweed and RHEL 9, and do now know the exact versions of Debian and Ubuntu that support OTLP.
You can also use the syslog-ng container image to test OTLP support in syslog-ng. You can find more information about the various images at https://www.syslog-ng.com/community/b/blog/posts/nightly-syslog-ng-container-images.
You need at least two hosts for experimentation, one acting as a server and another one as a client in OTLP communication. You can use the client host or a third host to feed the OTLP client machine with logs.
Basic configuration
On the server side, this is the simplest configuration. Just append to syslog-ng.conf or create a new configuration snippet under the /etc/syslog-ng/conf.d/ directory if the syslog-ng package on your OS supports it.
source s_syslog_ng_otlp {
syslog-ng-otlp(port(6515));
};
destination d_file {
file("/var/log/otel");
# file("/var/log/otel.json" template("$(format-json --scope everything)\n\n"));
};
log {source(s_syslog_ng_otlp); destination(d_file); };
The commented-out line is there for reference: if you want to take advantage of the syslog-ng-otlp() transferring most name-value pairs, you can use this file destination to check those.
The client side is also simple. It has a network source, a syslog-ng-otlp() destination, connected by a log statement:
source net {
network(transport(tcp) port(6666));
};
destination d_syslog_ng_otlp {
syslog-ng-otlp(url("172.16.167.170:6515"));
};
log {source(net); destination(d_syslog_ng_otlp); };
Obviously, in both cases, you should replace IP addresses and port numbers with those suitable for your own environment. Once you have reloaded syslog-ng, you are ready to do some testing.
Testing
Initial testing is simple: just send some log messages to the network source using loggen, the bundled testing and benchmarking tool of syslog-ng:
loggen -i -S -r 1000000 172.16.167.174 6666
If you are not yet familiar with loggen:
loggen –help-all
The main page is a bit outdated, as it still refers to balabit.com… The parameters used above:
-
-i -S: TCP connection
-
-r 1000000: try to send a million messages a second (the default, 1000, is not enough for performance testing even on a Raspberry Pi)
-
IP address and port
Afterwards, you can check the file /var/log/otel on the server if logs arrived as expected.
Note that unless you set the keep_hostname() to yes, logs will look as if they arrived from localhost. Using keep_hostname(yes) solves this problem for now. This behavior might change in the future.
Tuning
There are a number of additional parameters which became available in 4.6.0 and can be very useful:
-
workers()
-
batch-lines()
-
compression()
The number of workers() determines the number of parallel connections opened by syslog-ng while using the syslog-ng-otlp() driver. In my tests, not using the workers() parameter, the speed of syslog-ng-otlp was minimally slower than using ewmm() destination over a tcp() connection. However, when you set workers() to a value larger than one, its speed will be better than ewmm. Make sure that you do not set a larger value than the number of CPU cores, otherwise it can even be slower than using a single worker.
You should also experiment with batching. Setting batch-lines() to a larger value can improve performance. However, just like with workers, setting a value that is too high might have side effects. I do not have suggestions for exact numbers, because in my tests, effects were different with the number of workers, settings on the destination, if compression is enabled, and so on. Test which setting works the best in your environment.
Once you enable compressing with compression(yes), syslog-ng will use less bandwidth at the expense of a bit more CPU usage. However, if you pay for network traffic or your network is saturated, then enabling compression is still worth the extra CPU cycles.
What is next?
Sending logs without encryption is OK for testing. However, if you plan to use OTLP in production, you should also enable encryption.
OpenTelemetry protocol support is a pretty recent feature in syslog-ng, it is still under development. Please share your experiences with us on GitHub: https://github.com/syslog-ng/syslog-ng You can also find me on LinkedIn, Twitter and Mastodon.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.