UDP-based log collection is so last century. We had TCP-based log collection for decades and TLS encryption to secure connections. Still, UDP is in wide use, especially at large companies and industrial automation, where every change is slow. In most cases, UDP logging is used by networking devices, but sometimes it is just left there from ancient times and people are reluctant to change it. In either case, at higher message rates it can lead to performance problems and thus to message loss.
Originally, the udp() source of syslog-ng was single-threaded. That does not scale well with typical multi-core CPUs with slower cores. There are many tricks to enhance UDP performance in syslog-ng. Combining those with the udp-balancer() source of syslog-ng PE gives the most reliable solution.
Before you begin
While much of what I write here applies to syslog-ng OSE as well, the udp-balancer() is a feature specific to syslog-ng Premium Edition (PE). Syslog-ng PE is the commercial variant of syslog-ng, lacking some of the more experimental features of the open-source edition (OSE) and adding a few extra features mostly related to cloud and compliance. Or in this case: collecting UDP log messages at a high message rate. It has been available for quite some time now, but as usual, using the latest version is recommended. I did my tests using syslog-ng PE 7.0.25. Note that even though syslog-ng PE supports RHEL 7, udp-balancer() itself is not supported by that OS.
If you do not have syslog-ng PE yet, contact One Identity sales for a trial version at https://www.syslog-ng.com/products/log-management-software/
Why udp-balancer()?
Configuring an udp() source can be a frustrating experience. You have to consider the amount of traffic, how is it distributed among source hosts in time and proportion, the network card and even single core CPU performance. With the udp-balancer() of syslog-ng PE, you only need to configure it once and you can use it in just about any situation without caring about the details. The udp-balancer() source uses eBPF to distribute incoming log messages among multiple listeners. This means that no matter the number of log sources and the distribution of incoming log messages over time and hosts, syslog-ng PE can utilize the hardware efficiently while also reducing the chance of message loss.
UDP traffic: from minor to major
In the next few paragraphs, I’ll show you the different UDP logging scenarios and the configuration settings to consider. Even if you have some doubts now, by the end you will appreciate the simplicity of using the udp-balancer() source.
The basics
If you have a low UDP message rate, there is no need to tune anything, syslog-ng will handle it just fine. But if the message rate has some bursts or constantly over a few thousands messages a second, it is time to tune the Linux kernel and syslog-ng for UDP. For these moderate message rates, tuning buffers is usually enough by:
-
Changing the net.core.rmem_max sysctl value.
-
Increasing the value of the log_fifo_size() option.
-
Increasing the value of so-rcvbuf() at the UDP source to match rmem_max.
This way, incoming UDP packages can be buffered by the Linux kernel and can be kept there while syslog-ng works through incoming messages.
For more details about these values, check the syslog-ng PE documentation at https://www.syslog-ng.com/technical-documents/doc/syslog-ng-premium-edition/7.0.25/collecting-log-messages-from-udp-sources
Few logs from many hosts
Once you have even higher message rates (higher here depends also on the hardware), things get complicated. You’ll start to lose log messages, as a single thread cannot keep up with the flood of incoming logs. If you collect logs from many hosts, and none of them send logs at a high message rate, there is still some hope in the form of so-reuseport(). Using this option, syslog-ng can open multiple listeners on the same UDP port, and the kernel distributes incoming packages among listeners based on source IP addresses. Of course, it only works with many sources that do not have too high message rates. As soon as there is a host with high message rate, messages cannot be distributed evenly among listeners. The balance is lost, and message loss goes up.
When a single UDP port cannot handle the load anymore, you can open additional UDP sources in syslog-ng. But doing so requires extra configuration on the server side and also re-configuring part of your clients to direct them to the other ports.
Lots of logs from a few hosts
The most typical situation for UDP logging is that there are many logs from just one or a few hosts, for example a router or a firewall trying to document each and every connection going through the device. As so-reuseport() distributes packages based on the source IP address, that feature cannot help here.
This is where the udp-balancer() source of syslog-ng PE can come handy. It is using eBPF to distribute incoming UDP packages evenly among listeners. You do not have to open multiple ports and reconfigure your clients to make use of them.
Best of all: using the udp-balancer() source of syslog-ng PE is not specific to high-performance log collection. You can use it in any situation when you collect log messages over UDP.
Testing
Ideally, I would do testing on a 10GBit network, but that is not (yet) available in my home network. That is how I ended up testing from the localhost with a virtually unlimited network connection.
I configured a 256MB buffer:
sysctl -w net.core.rmem_max=268435456
And used the following syslog-ng configuration with matching buffer sizes. It is based on the default syslog-ng PE configuration, but I added a new source (with the traditional and the new UDP source) and a new file destination as well.
@version: 7.0
#Default configuration file for syslog-ng.
#
# For a description of syslog-ng configuration file directives, please read
# the syslog-ng Administration Guide at:
#
# https://support.oneidentity.com/syslog-ng-premium-edition/technical-documents
#
@include "scl.conf"
options {
stats_freq(0); log_fifo_size(4190);
};
######
# sources
source s_local {
# message generated by Syslog-NG
internal();
system();
monitoring_welf();
};
######
# destinations
destination d_messages { file("/var/log/messages"); };
log {
source(s_local);
destination(d_messages);
};
source s_net {
udp(port(514) so-rcvbuf(268435456) );
udp-balancer(
port(2222)
so-rcvbuf(268435456)
);
};
destination d_fromnet {
file("/var/log/fromnet");
};
log {source(s_net); destination(d_fromnet);};
Testing was done using the loggen utility of syslog-ng. In case of syslog-ng PE, it is installed to a non-standard location. Either add it to your PATH or call it using the full path name:
/opt/syslog-ng/bin/loggen -r 140000 --active-connections=8 -i -D localhost 2222
If you used loggen before, you will notice the number after the -r (maximum message rate) option. With TCP connections, we normally use insanely large values as it builds out connections and regulates itself. UDP does not use connections, so loggen generates log messages as fast as it can.
The --active-connections option tells loggen how many threads to use to send log messages. Note that the -r option is multiplied by this number.
The rest of the options tell loggen to send logs to localhost on port 514 over UDP.
You are now ready to experiment. Start with small numbers and increase until you see considerable message loss. Or start with large numbers and keep decreasing until message loss disappears or gets to a tolerable level.
Note: do not forget to remove the log file and restart syslog-ng between test runs. Otherwise, suddenly you will count a lot more logs than what is actually sent :)
On my host, the above loggen command resulted in the optimal load. What does it mean? When I compared the number of messages loggen sent with the number of lines in the log file, the difference was less than a hundred. Depending on your hardware, the values might be slightly higher or much lower.
As a next step, send the same number of logs to port 514, the regular UDP source. You will see that only a fraction of the log messages are processed and saved by syslog-ng. You can now play with “-r” again to find the value where message loss drops to an acceptable value.
Summary
A picture is worth a thousand words, so let me summarize this blog with two screenshots. One was taken when using the regular udp() source of syslog-ng, the other one when using the udp-balancer() source. In the first case, one CPU core is fully loaded, while the rest of the cores are idle. In the second case, the load is distributed near evenly.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.