Last time, we had an overview of the syslog-ng disk-buffer. This time, we dig a bit deeper and take a quick look at how it works, and a recent major change that helped speed up the reliable disk-buffer considerably.
You can read my disk-buffer overview at https://www.syslog-ng.com/community/b/blog/posts/when-not-to-use-the-syslog-ng-disk-buffer
Before you begin
The disk-buffer landed in syslog-ng Open Source Edition with version 3.8.1, quite some time ago. It was available in the commercial variant of syslog-ng (syslog-ng Premium Edition) even earlier. The major speed-up arrived in syslog-ng in version 3.34.1, barely half a year ago, and there have been improvements ever since. This is why I recommend using the latest version available, or version 3.35.1 at least. If your operating system of choice comes with an earlier syslog-ng version, you can find links to up-to-date packages on the syslog-ng 3rd party repository page at https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx
How does it work?
Both the reliable and the non-reliable disk-buffer have the same structure now. There is a memory-buffer before and after the disk-buffer. Messages arrive into the first one, called “output queue”. If the destination is reachable and is fast enough to collect messages, then log messages are transferred there directly from the “output queue”.
Non-reliable:
Reliable:
The difference between reliable and non-reliable disk-buffering is how the disk-buffer file is handled.
When using the non-reliable disk-buffer, log messages are only persisted to the disk-buffer file if the destination is unreachable or cannot handle the load. If there is a power failure or syslog-ng is not properly terminated, then logs in the buffers are lost. No logs are lost when syslog-ng is properly shut down, as the contents of memory-buffers are saved.
When using the reliable disk-buffer, all log messages are written to disk immediately. As a result log messages are not lost, even if someone accidentally pulls the power cord.
What changed?
Originally, there were much larger differences between the two disk-buffer methods. The reliable disk-buffer did not have that extra memory-buffer before the disk-buffer, but wrote all messages directly to the disk-buffer file instead. Once the log was saved, it was read back from the disk. Reading has a heavy I/O overhead, while re-creating the message from data on disk can impact CPU usage. We cannot spare writing log messages to disk. This is required for reliable disk-buffering. However, using the contents of the output queue (instead of reading data from the disk-buffer and reconstructing the message) can save us lots of I/O and CPU usage.
Risks
Configuring the various buffers related to the disk-buffer can be a bit tricky. Some values need to be configured as the number of messages, while others as the number of bytes. The larger memory-buffers you use, the faster sending messages gets. However, if you use the non-reliable disk-buffer method, the larger memory-buffers also mean greater risk of message loss in case of abnormal termination. When using the reliable disk-buffer method, sizing the mem-buf-size() is critical. In this case, it is better to overestimate the size to avoid bottlenecks. You can find the parameter names on the graphs above, and read more about how to configure the various values in the technical documentation: https://www.syslog-ng.com/technical-documents/list/syslog-ng-open-source-edition/
What is next?
In the next disk-buffer blog, I will explain why disk-buffer queue files can stay huge even when they are empty.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.