One of the returning questions I receive is how many log messages can a given hardware handle. My typical answer is that it depends on the configuration. I have now an answer, or rather a tool to answer your question sngbench.sh. It is a shell script that runs from localhost and uses loggen, the bundled benchmarking and testing tool of syslog-ng. It comes with two configurations: a performance-optimized and a realistic one. You are also free to extend sngbench with your own configurations.
History
The original idea of this script came many years ago, when I got access to various interesting platforms, such as ARM, POWER, RiscV, and even MIPS. To compare these, I wrote a quick and dirty script. The main goal was CPU comparison, so the loggen runs were short to avoid disk I/O.
Many people got interested in it, but the script was ugly and difficult to use. So, with the help of István Szántai, the code was refactored, and developed into something easy to read and use. It is no longer limited to short runs, values are not hard-coded, and it is a single script run, instead of running scripts four different times.
What is it good for?
The script and the included configurations can help you compare how syslog-ng performs on various hardware and operating systems. As you can also extend it with your own configurations, it can also be used to measure how various configuration changes affect performance.
One of the included configurations is focused on performance, that is how far syslog-ng scales with a simple tuning configuration. The other configuration is more realistic. It is a typical Linux distro configuration, taken from openSUSE, and modified only minimally. It does lots of message parsing and filtering.
Loggen connects to syslog-ng using TCP connections using localhost. There are multiple test runs, starting with just a single connection to a single port, and up to four loggen clients running in parallel using 128 threads each to send logs to four different ports on syslog-ng. Of course, you are free to change loggen parameters as well, not only the syslog-ng configurations.
Litmitations
There are quite a few limitations you have to be aware of. First of all, tests are run on the localhost. It makes preparation easy, but it also means that limitations of the network card are not considered. A gigabit network connection (or even a 10GBit one) might be more limiting than disk or CPU performance. Also, as loggen runs on the same host as syslog-ng, they compete for CPU resources.
Loggen is a synthetic benchmark. The log messages it sends are practically the same and pushed at full speed all the time towards syslog-ng. In real life, the length and content of log messages are different, and log sources range from a few logs each minute to hundreds of thousands of messages each second.
This is why maximum values shown by the script are more theoretical, than practical. Still, these values are useful, as they allow you to compare various hardware, operating systems and configurations.
How to get started?
The most important prerequisite is the up-to-date bash. There are a few more, which are available on most Linux systems by default, and easy to install on the Mac or FreeBSD. The code and configurations are available on GitHub:
git clone https://github.com/czanik/sngbench
You can find the documentation there as well. I do not want to list it here.
Results
Each script run creates a new sub-directory. All output from loggen runs is collected with some basic information about the host. The most important file in the directory is results.csv. It has the following format:
czplaptop:~/sngbench/out/2023-05-24_16-15-48 # cat results.csv czp_distro,I_10_i_r_3000000_activeconnections_10_S_C_127.0.0.1_514,1,324365.31,324208.94,346084.93 czp_distro,I_10_i_r_3000000_activeconnections_128_S_C_127.0.0.1_514,1,545093.87,545314.80,568396.64 czp_distro,I_10_i_r_3000000_activeconnections_1_S_C_127.0.0.1_514,1,164845.46,147174.57,158488.40 czp_distro,I_10_i_r_3000000_activeconnections_32_S_C_127.0.0.1_514,1,154310.87,87772.86,144031.75 […] czp_maxperf,I_10_i_r_3000000_activeconnections_32_S_C_127.0.0.1_514,1,143676.41,132700.81,141836.21 czp_maxperf,I_10_i_r_3000000_activeconnections_32_S_C_127.0.0.1_515,2,149104.33,129430.51,149507.70 czp_maxperf,I_10_i_r_3000000_activeconnections_32_S_C_127.0.0.1_516,3,143136.92,135242.09,143571.29 czp_maxperf,I_10_i_r_3000000_activeconnections_32_S_C_127.0.0.1_517,4,151772.81,136636.63,143685.36 czp_maxperf,I_10_i_r_3000000_activeconnections_50_S_C_127.0.0.1_514,1,524238.95,447613.18,210249.44 czplaptop:~/sngbench/out/2023-05-24_16-15-48 #
Lines start with the name of the configuration file, the loggen parameters, and the message rate from three test runs.
Further processing and visualization is your choice. The GIT repo also contains a sample run with a LibreCalc spreadsheet averaging the three test runs and adding values for tests where multiple loggen clients run together. It allows you to create nice graphs like this:
What is next?
If you have ideas to improve the script, open a pull request. If you have some interesting measurement results, please share those as well.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.