It is easy to over-complicate log management. Almost all departments in a company need to log messages for their daily activities. However, installing several different log management and analysis systems in parallel is a nightmare both from a security and an operations perspective and wastes many resources. You cannot always reduce the number of log analysis systems, but you can reduce the complexity of log management. Let me show you, how.
The problem
Even just half a decade ago most companies did not want to deal with collecting log messages. If they did, it was all due to following compliance regulations. Nowadays we have a different problem. Everybody wants log messages. We have a data-driven economy, so it is not just a security analyst trying to reproduce steps of a breach, but marketing also wants to analyze web server logs, HR wants to see if remote workers are working, and so on.
Of course, all departments have their own requirements on what logs they want to collect and what kind of reports and alerts they want to have from those logs. Often it is a bit like the Wild West, as all departments want to have their own log analysis platforms accompanied by its own log management system. Why is it a problem? Because installing multiple log management software means that:
-
security needs to check multiple client software.
-
operations needs to install multiple client software.
-
more disk space, more network traffic is used.
In other words: a lot of complexity, which results in more human and computing resources needed.
The solution
Reducing the number of different log analysis platforms is not easy, but sometimes possible. For example, by using different dashboards of the same Splunk or Elasticsearch database. However, reducing the complexity of log management is easier than you think. You can collect log messages using a single software to a central location. Once all logs are available in a single location, you can process them, or even solve the long-term archiving of log messages and forward log messages to various log analysis software based on their requirements.
There are many advantages of this approach for everyone in your organization:
-
Security needs to check only a single software instead of several different ones.
-
Operations needs to install a single software on clients and servers instead of several different ones.
-
Less network or computing resources are used.
-
Instead of trying to solve long-term log storage in several different log analysis software, you can do it in a single location at the log management level.
As you can see, you can drastically reduce the complexity of log management and save on both human and computing resources.
The tools
One of the best tools to create a dedicated log management layer is syslog-ng. Thousands of servers run syslog-ng and millions of computers on the client side. However, not many people are aware that syslog-ng is not a single software, but a product line that consists of the following:
-
syslog-ng Open Source Edition (OSE)
-
syslog-ng Premium Edition (PE)
-
syslog-ng store box (SSB)
All of them collect log messages from many sources, parse the logs, enrich them, filter them, and forward them to the various log analysis solutions in use at your organization. Let me introduce you to each of the syslog-ng variants.
Syslog-ng Open Source Edition
Syslog-ng OSE is what most people know as syslog-ng. It is a Swiss army knife of logging that needs to be configured in a text-based configuration file and has a large number of input, processing and output possibilities.
Syslog-ng OSE is where most of the sylog-ng development is done and it includes quite a few experimental features as well. Syslog-ng OSE is not officially supported by One Identity, but there is nothing to worry about. Before syslog-ng is extended with a new feature, thousands of tests ensure that the existing functionality is not affected. Developers support syslog-ng at a best effort level and there is also large community around syslog-ng OSE.
As it is open source, it is part of most Linux distributions and it is available in the ports system of most BSD variants. Various unofficial repositories are also available with the latest syslog-ng versions for major Linux distributions. You can also compile and even fix it yourself if necessary. The syslog-ng code is portable, which means that it easily compiles not only on x86, but also on ARM, POWER, RiscV and even on completely unknown architectures.
Syslog-ng Premium Edition
Syslog-ng PE is what our sales and commercial customers refer to as syslog-ng. It is based on syslog-ng OSE sources, but without the experimental features. It only includes the thoroughly tested parts of OSE and adds a number of PE-exclusive features related to compliance (for example: encrypted, time stamped log storage) and features related to cloud (for example: Google Pub/Sub destination).
Just like OSE, PE also has a text-based configuration, but it comes with Enterprise level support. You should use PE if you need one of the compliance- or cloud-related features exclusive to PE or if you need more than just best effort support. Currently only x86 enterprise Linux distributions are supported on the server side, and there is also an agent for Windows.
Syslog-ng store box
Unlike the other two offerings, the syslog-ng store box is an appliance. This appliance can be either physical or virtual and it comes with everything ready to use. It has syslog-ng PE at its core, but configured through a web-based interface. It also provides a search interface for the logs and provides you with full log life cycle management automation, including backup. Most of the syslog-ng PE functionality is supported, but not everything, and it is less flexible compared to what can be done using the text-based configuration. However, its main advantage is that instead of spending hours trying to understand how it works, once you launch the appliance, it is ready to collect log messages within minutes.
If you are a Windows user, or simply do not have the time to learn new technologies, or configure and integrate log management yourself from scratch, SSB is a good choice for log management. The Google-like speed of returning search results in terabytes of logs is an added bonus.
Getting syslog-ng
If you do not have syslog-ng yet, it is time to try it. The syslog-ng open source edition is available in most Linux distributions and BSD variants. If you are interested in one of the enterprise variants of syslog-ng, you can ask for a trial version on the syslog-ng website at https://www.syslog-ng.com/trials/