Creating a dedicated log management layer

Event logging is a central source of information both for IT security and operations, but different teams use different tools to collect and analyze log messages. The same log message is often collected by multiple applications. Having each team using different tools is complex, inefficient and makes systems less secure. Using a single application to create a dedicated log management layer independent of analytics instead, however, has multiple benefits.

Using syslog-ng is a lot more flexible than most log aggregation tools provided by log analytics vendors. This is one of the reasons why my talks and blogs focused on how to make your life easier using its technical advantages. Of course, I am aware of the direct financial benefits as well. If you are interested in that part, talk to my colleagues on the business side. They can help you to calculate how much you can save on your SIEM licenses when syslog-ng collects log messages and ensures that only relevant messages reach your SIEM and only at a predicatively low message rate. You can learn more about this use case on our Optimizing SIEM page.

In this blog, I will focus on a third aspect: simplifying complexity. This was the focus of many of my conference discussions before the COVID-19 pandemic. If we think a bit more about it, we can see that this is not really a third aspect, but a combination of the previous two instead. Using the flexibility of syslog-ng, we create a dedicated log management layer in front of different log analytics solutions. By reducing complexity, we can save in many ways: on computing and human resources, and on licensing when using commercial tools for log analysis as well.

Back to basics

While this blog is focusing on how to consolidate multiple log aggregation systems that are specific to analytics softwares into a common log management layer, I also often see that many organizations still do not see the need for central log collection. So, let’s quickly jump back to the basics: why central log collection is important. There are three major reasons:

• Convenience: a single place to check logs instead of many.

• Availability: logs are available even when the sender machine is down or offline.

• Security: you can check the logs centrally even if a host was breached and logs were deleted or falsified locally.

The client-relay-server architecture of syslog-ng can make sure that central logging scales well, even for larger organizations with multiple locations. You can learn more about relays at https://www.syslog-ng.com/community/b/blog/posts/what-syslog-ng-relays-are-good-for

Reducing complexity

Collecting system logs with one application locally, forwarding the logs with another one, collecting audit logs with a different app, buffering logs with a dedicated server, and processing logs with yet another app centrally means installing several different applications on your infrastructure. This is the architecture of the Elastic stack, for example. Many others are simpler, but still separating system log collection (journald and/or one of the syslog variants) and log shipping. This is the case of Splunk forwarder and many of the different Logging as a Service agents. And on top of that, you might need a different set of applications for different log analysis software. Using multiple software solutions makes a system more complex, difficult to update and needs more computing, network and storage resources as well.

All these features can be implemented using a single application, which in the end can feed multiple log analysis software. A single app to learn and to follow in bug & CVE trackers. A single app to push through the security and operations teams, instead of many. Less resources needed both on the human and technical side.

Implementing a dedicated log management layer

The syslog-ng application collects logs from many different sources, performs real-time log analysis by processing and filtering them, and finally, it stores the logs or routes them for further analysis.

In an ideal world, all log messages come in a structured format, ready to be used for log analysis, alerting or dashboards. But in a real world, only part of the logs fall into this category. Traditionally, most of the log messages come as free format text messages. These are easy to be read by humans, which was the original use of log messages. However, nowadays logs are rarely processed by the human eye. Fortunately, syslog-ng has several tools to turn unstructured (and many of the structured) message formats into name-value pairs, and thus delivers the benefits of structured log messages.

Once you have name-value pairs, log messages can be further enriched with additional information in real-time, which helps responding to security events faster. One way of doing that is adding geo-location based on IP addresses. Another way is adding contextual data from external files, like the role of a server based on the IP address or the role of the user based on the name. Data from external files can also be used to filter messages (for example, to check firewall logs to determine whether certain IP addresses are contained in various black lists for malware command centers, spammers, and so on).

Logging is subject to an increasing number of compliance regulations. PCI-DSS or many European privacy laws require removing sensitive data from log messages. Using syslog-ng logs can be anonymized in a way that they are still useful for security analytics.

With log messages parsed and enriched, you can now make informed decisions where to store or forward log messages. You can already do basic alerting in syslog-ng, and you can receive critical log messages on a Slack channel. There are many ready-to-use destinations within syslog-ng, like Kafka, MongoDB or Elasticsearch. Also, you can easily create your own custom destination based on the generic network or HTTP destinations, and using templates to log in a format as required by a SIEM or a Logging as a Service solution, like Sumo Logic.

What is next?

Many of these concepts were covered before in earlier blogs, and the individual features are covered well in the documentation. If you want to learn more about them and see some configuration examples, join me at the Pass the SALT conference, where among many other interesting talks, you can also learn more in depth about creating a dedicated log management layer.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.