UNITE is the partner and user conference of One Identity, the company behind syslog-ng. This time the conference took place in Phoenix, Arizona where I talked to a number of American business customers and partners about syslog-ng. They were really enthusiastic about syslog-ng and emphasized two major reasons why they use syslog-ng or plan to introduce it to their infrastructure: syslog-ng allows them to reduce the log data volume and greatly simplify their infrastructure by introducing a separate log management layer.
Reduce
Log messages are very important both for the operation and security of a company. This is why you do not just simply store them, but feed the log messages to SIEM and other log analysis systems that create reports and actionable alerts from your messages.
Applications can produce tremendous amount of log data. This is a problem for SIEM and other log analysis systems for two major reasons:
-
hardware costs, as the more data you have the more storage place and processing power you need to analyze the data
-
licensing costs, as most analysis platforms are priced on data volume
You can easily reduce message volume by parsing and filtering your log messages and only forwarding the logs for analysis which are really necessary. Many people started to use syslog-ng just for this use case, as it is really easy to create complex filters using syslog-ng.
This is why I was surprised to learn about another approach: sending all log messages, but not whole messages, only the necessary parts. This needs a bit of extra work, as you need to figure out which part of the log message is used by your log analysis application. But once you are ready with your research, you can easily halve the log messages, or in some special cases even reduce the message volume by 90%.
Some examples are:
-
Reading the name-value pairs from the systemd journal, but forwarding only selected name-value pairs.
-
Parsing HTTP access logs and forwarding only those columns which are actually analyzed by your software.
The syslog-ng application has powerful parsers to segment the log messages to name-value pairs, after which you can use templates and template functions of syslog-ng for such selective log delivery.
If your log analysis infrastructure is already in place, it is still worth to make the switch to syslog-ng and reduce your log volume using these techniques. You can use the current log analysis infrastructure for a lot longer time without having to expand it with further storage and processing power.
Simplify
Most SIEM and log analysis solutions come with their own client applications to collect log messages. So, why bother installing a separate application from yet another vendor to collect your log messages? Installing syslog-ng as a separate log management layer does not actually complicate your infrastructure, but rather simplifies it:
-
No vendor lock-in: replacing your SIEM is pain free and quick, as you do not have to replace all the agents as well
-
Operations, security and different teams of the company use different software solutions to analyze log messages: instead of installing 3-4 or even more agents, you only install one that can deliver the required log messages to the different solutions.
When you collect log messages to a central location using syslog-ng, you can archive all of the messages there. If you add a new log analysis application to your infrastructure, you can just point syslog-ng at it and forward the necessary subset of log data there.
Life at both security and operations in your environment becomes easier, as there is only a single software to check for security problems and distribute on your systems instead of many.
What is next?
If you are on the technical side, I recommend you reading two chapters from the syslog-ng documentation:
These explain you how you can reformat your log messages using syslog-ng, giving you a way to reduce your data volume significantly by including only necessary name-value pairs.
If you want to learn more about this topic, our Optimize SIEM white paper explains it in more depth.
The open source version of syslog-ng is part of most Linux distributions, but packages might be outdated. For up-to-date packages check the 3rd party binaries page for information.
If you need commercial level support and help in integrating syslog-ng to your environment, start an evaluation of syslog-ng Premium Edition.
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.
