Throttling log messages

One of the main advantages of syslog-ng is that it is high performance and low on resource usage. Why throttle the messages then? There are three main reasons – licensing, performance, and bandwidth – all outside of syslog-ng. From this blog, you can learn how to identify use cases for the throttling of log messages, read about a possible drawback, and finally get a sample configuration.

What is throttling?

Most destinations in syslog-ng support the throttle() option. This options sets the maximum number of messages sent to a destination in a second. In order to avoid message loss, it is best to combine throttling with the disk buffer. If there is a larger peak in message rate, the disk buffer can temporarily store messages.

Why throttling?

Depending on the exact hardware and configuration, syslog-ng can collect, process, and forward hundreds of thousands of messages each second. On the other hand, not all destinations where syslog-ng sends logs can handle this message rate.

Licensing

Log messages are often forwarded to Security Information and Event Management (SIEM) systems. Some of these are licensed based on message rates. The targeted message rate is usually well above the average message rate, but still it is possible that due to some unforeseen events, the message rate peaks at a much higher value. Using the throttle() option of syslog-ng, you can make sure that all messages arrive to your SIEM while staying within license boundaries.

Performance

In-depth analysis of log messages – as done by SIEM and other software – requires lots of resources and it is usually a slow process. This means that those software can work with a relatively low message rate. If you know the limits of your log analysis software, you can set the throttle limit in syslog-ng making sure that all of your log messages are analyzed.

Bandwidth

Sending messages over a network takes bandwidth. At peak message rates, you can easily saturate even a Gigabit network with log messages. You might have a network where it is critical that you have enough bandwidth left for some of your applications. Knowing your average message size (or counting with the maximum message size) and using the throttle() option of syslog-ng, you can make sure that syslog-ng never uses more than a given amount of bandwidth.

Drawbacks

Throttling means that if the configured message rate is reached, messages start to queue up in memory and on disk for seconds or minutes before reaching their destination. This side-effect can be problematic in some situations where staying (near) real-time is critical. You can also reduce message rates with more aggressive filtering, but that might not resolve all problems that throttling can, as peak message rate can still go up.

Configuring throttle

The example configuration below shows a network source and a network destination, as throttling is usually configured in a relay. It also makes testing easier.

source s_network {

        network(

            ip("0.0.0.0")

            port("514")

            transport(tcp)

            ip-protocol(4)

        );

};

destination d_network {

    network(

      "172.16.167.137"

      port(514)

      throttle(500)

      disk-buffer(

        disk-buf-size(10000000)

        reliable(no)

        dir("/tmp/disk-buffer")

      )

    );

};

log {

  source(s_network);

  destination(d_network);

  flags(flow-control);

};

In our case, the destination is the most important part. Here you can see the IP address and port number where log messages are sent. The throttle() option is configured to send a maximum of 500 messages a second. Finally, there is a huge disk buffer to hold messages temporarily when necessary.

Testing

If you used the above configuration, you can easily test it using loggen. Just make sure that there is also a receiving end where you store all incoming logs to a file.

For example:

destination d_test { file("/var/log/test"); };

source s_network {

        network(

            ip("0.0.0.0")

            port("514")

            transport(tcp)

            ip-protocol(4)

        );

};

log {

  source(s_network);

  destination(d_test);

};

On the receiver machine, tail this file, for example:

tail -f /var/log/test

On the sending side, start loggen with the following options:

loggen -r 10000 -i -S localhost 514

You should see that messages are still arriving on the receiver machine long after loggen printed its statistics on the sender machine.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/balabit/syslog-ng. On Twitter, I am available as @PCzanik.