Improved log collection over UDP

“I'd tell you the joke about UDP, but you might not get it.”

The old joke above perfectly summarizes UDP. There is no guarantee (frankly, not even a real effort) that data sent over UDP ever reaches the receiving end. Still, a surprisingly large number of syslog(-ng) users still keep using UDP as their transport protocol. The introduction of the so-reuseport() option for the UDP source in syslog-ng 3.19 is for those people who, for policy or other reasons, can not switch to TCP logging but still want to make UDP log reception as reliable as possible.

Up until recently, incoming log messages over UDP were handled by a single thread within syslog-ng. While there are some buffering options in the kernel and in syslog-ng (see so-rcvbuf() and others in the documentation), you still had to define multiple UDP sources and load-balance log messages on the sender side to achieve better reliability.

Now, however, when the so-reuseport() option is enabled, the kernel allows multiple UDP sockets to be bound to the same port and the kernel load-balances incoming UDP datagrams to the sockets. The sockets are distributed based on the hash of (srcip, dstip, srcport, dstport), so the same listener should be receiving packets from the same endpoint.

Here you can see a UDP source where the same port is bound four times:

source s_udp {
 udp(so-reuseport(1) port(2000) persist-name("udp1"));
 udp(so-reuseport(1) port(2000) persist-name("udp2"));
 udp(so-reuseport(1) port(2000) persist-name("udp3"));
 udp(so-reuseport(1) port(2000) persist-name("udp4"));
};

I did some load tests using the loggen utility of syslog-ng. At the end of the process, I compared the number of messages sent with the number of messages actually saved by syslog-ng. Regardless of how many loggen instances sent logs to syslog-ng, without using so-reuseport() the amount of logs actually saved was practically constant.

However, using the configuration above, the situation was much better: the amount of logs saved was in close correlation with the amount of logs sent from the loggen clients.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/balabit/syslog-ng. On Twitter, I am available as @PCzanik.