There are multiple ways in syslog-ng to limit message rate. The throttle() option of syslog-ng destinations tries to make sure that all messages are delivered without exceeding a specified message rate. The rate-limit() filter introduced in syslog-ng 3.36 drops surplus log messages, making sure that a processing pipeline or destination is not overloaded with log messages.
Before you begin
If you just need the throttle() option in a syslog-ng destination, then you most likely do not need to install the latest syslog-ng version. Even RHEL 7 has version 3 of syslog-ng available in EPEL. However, to use the rate-limit() filter, you need to use at least syslog-ng 3.36. While it is not yet available in most Linux distributions, there is a good chance that you can find it in a third-party repository: https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx
Using throttle()
The throttle() option of various syslog-ng destinations makes sure that the message rate is throttled down to a value acceptable by the specific destination. As in-memory buffers can fill up quickly, you are advised to combine the throttle() option with a disk-buffer. This way, no messages are lost if there is a sudden spike in message rate. Some destinations are throttled by default, like discord(), where a rate of more than a couple messages per second is considered flooding.
Adding throttling to a destination is easy, just do not forget to also add a disk-buffer. Otherwise, there is a good chance for message loss. Here is a simple example allowing 100 messages a second over a TCP connection, combined with a larger disk-buffer.
destination d_throttled {
network(
"127.0.0.1"
port(3333)
throttle(100)
disk-buffer(
mem-buf-length(10000)
disk-buf-size(2000000)
reliable(no)
dir("/tmp/disk-buffer")
)
);
};
Using rate-limit()
The rate-limit() filter was originally called throttle(), but it was renamed to rate-limit() to avoid confusion. This is because there is a fundamental difference between the throttle() option of destinations and the rate-limit() filter: here, the message loss is intended. The rate-limit() filter can be used as a precaution: ensuring that a specific resource cannot be flooded with messages. You can set a simple limit: in this case, any message above the specified rate is simply discarded. There is also a possibility to configure a template using macros. For example, using the $HOST macro, the message rate is calculated for each host separately. This way, a single host cannot overload a message pipeline with a peak of messages, as messages above the rate-limit() are discarded.
filter f_rate_limit {
rate-limit(
template("$HOST")
rate(5000)
);
};
If you put this filter in the log path on a relay forwarding log messages from five hosts to a central syslog-ng server, it makes sure that no hosts can send more than 5000 messages a second. Surplus log messages are discarded. In this case, this means that the message rate does not exceed 25000 messages a second.
Testing
Testing the template() part of the rate-limit() filter is not easy, but testing the rate-limit() filter can easily be done using a network source and loggen. Append this configuration to syslog-ng.conf or place it with a .conf extension in the /etc/syslog-ng/conf.d, if using it is supported on your system.
source s_tcp {
tcp(port(514));
};
filter f_rlimit {
rate-limit(rate(100));
};
destination d_file {
file("/var/log/myfile");
};
log {
source(s_tcp);
filter(f_rlimit);
destination(d_file);
};
Once you reloaded syslog-ng, you should have a new tcp() source listening, and logs written to a file. Only 100 log messages are written to the file a second, the rest are discarded. Loggen sends around a thousand messages a second by default:
czplaptop:/etc/syslog-ng/conf.d # loggen -i -S localhost 514 count=1803, rate = 907.78 msg/sec count=2255, rate = 903.76 msg/sec count=2711, rate = 911.80 msg/sec [...] count=9959, rate = 899.70 msg/sec average rate = 995.62 msg/sec, count=9959, time=10.0028, (average) msg size=256, bandwidth=248.90 kB/sec czplaptop:/etc/syslog-ng/conf.d # wc -l /var/log/myfile 1005 /var/log/myfile
As you can see, 9959 messages were sent in 10.0028 seconds, but only 1005 were written to disk, which means that the limit on message rate was kept perfectly.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.