This is the 2017 edition of my most popular blog about syslog-ng web-based graphical user interfaces (web GUIs). Many things have changed in the past few years. In 2011 only a single logging as a service solution was available, now I regularly run into yet another one. Also, the number of logging-related GUIs is growing, so I will mostly focus on syslog-ng based solutions.
Centralized logging of events has been an important part of IT for many years. It is more convenient to browse logs in a central location rather than viewing them on individual machines. Central storage is also more secure. Even if logs stored locally are altered or removed, you can still check the logs on the central log server. Compliance with different regulations also makes central logging necessary.
System administrators often prefer to use the command line. Utilities such as grep and AWK are powerful tools, but complex queries can be completed much faster with logs indexed in a database and a web interface. In the case of large amounts of messages, a web-based SQL solution is not just convenient, it is a necessity. With thousands of incoming messages per second, the indexes of log databases still give Google-like response times even for the most complex queries, while traditional text-based tools are not able to scale as efficiently.
Logging as a Service (LaaS)
A couple of years ago Loggly was the pioneer of logging as a service (LaaS). Today there are many other LaaS providers (Papertrail, Logentries, Sumo Logic, and so on) and syslog-ng works perfectly with these. On the other hand it is better to think twice before relying only on a cloud service. LaaS needs a continuous network connection to avoid losing log messages, and high speed uplink network connection if you have more than just a few events per second. For logging applications that are already in the cloud this is less of a concern, but fast and reliable network connections are quite expensive and should be considered when looking for a log management solution for your local logs.
Structured fields and name-value pairs in logs are increasingly important, as they are easier to search and easier to create meaningful reports from. The more recent IETF RFC 5424 syslog standard supports structured data, but it is still not in widespread use.
People started to use JSON embedded into legacy (RFC 3164) syslog messages. The syslog-ng application can send JSON-formatted messages, for example, you can convert the following messages into structured JSON messages:
- RFC5424-formatted log messages
- Windows Eventlog messages received from the syslog-ng Agent for Windows application
- Name-value pairs extracted from a log message with PatternDB or the CSV parser.
Loggly and other services can receive JSON-formatted messages, and make them conveniently available from the web interface.
Some non-syslog-ng-based solutions
Before going on with solutions with syslog-ng at their heart, I would like to say a few words about the others, some which were included in the previous edition of the blog.
LogAnalyzer from the makers of Rsyslog was a simple, easy to use PHP application a few years ago. While it has developed quite a lot, recently I could not get it to work with syslog-ng. Some of the popular monitoring software have syslog support to some extent, for example, Nagios, Cacti and several others. I have tested some of these, I have even sent patches and bug reports to enhance syslog-ng support, but syslog is not the focus here, just one of the possible inputs.The ELK stack (Elasticsearch + Logstash + Kibana) and Graylog2 have become popular recently, but they have their own log collectors instead of syslog-ng, and syslog is just one of many log sources. Syslog support is quite limited both in performance and protocol support. They recommend the use of file readers for collecting syslog messages, but that increases complexity as it is an additional software on top of syslog(-ng) and filtering still needs to be done on the syslog side. Note, that syslog-ng can send logs to Elasticsearch natively which can greatly simplify your logging architecture.
Collecting and displaying metrics data
You can collect metrics data using syslog-ng. Examples include netdata or collectd. You can send the collected data to Graphite or Elasticsearch. Graphite has its own web interface, while you can use Kibana to query and visualize data collected to Elasticsearch. Another option is to use Grafana. Originally it was developed as an alternative web interface to Graphite databases, but now it can also visualize data from many more data sources, including Elasticsearch. It can combine multiple data sources to a single dashboard and provides fine grained access control.
Syslog-ng based solutions
As promised at the beginning, I will focus on syslog-ng based solutions. While every software described below is originally based on syslog-ng Open Source Edition (except for Balabit’s own syslog-ng Store Box (SSB)), there are already some large scale deployments also with syslog-ng Premium Edition as syslog server.
- The syslog-ng application and SSB focus on generic log management tasks and compliance
- ELSA serves the need of network security professionals
- LogZilla focuses on logs from Cisco devices
- Recent syslog-ng releases are also able to store log messages directly into Elasticsearch, a distributed, scalable database system popular in DevOps environments, which enables the use of Kibana for analyzing log messages.
Benefits of using syslog-ng PE with these solutions include the logstore, a tamper-proof log storage (even if it means that your logs are stored twice), binaries for over 50 platforms – including Windows – and vendor support.
There is a MongoDB based web GUI for syslog-ng, but it is not actively maintained anymore. Still, if you use MongoDB for all of your business data, this software might provide a base for your own log viewing software, as the source is available on GitHub: https://github.com/algernon/mojology
LogZilla is the commercial reincarnation of one of the oldest syslog-ng web GUIs: PHP-Syslog-NG. It provides the familiar user interface of its predecessor, but also includes many new features. The user interface supports Cisco Mnemonics, extended graphing capabilities, and e-mail alerts. Behind the scenes, LDAP integration, message de-duplication, and indexing for quick searching were added for large datasets.
Over the past years it received many small improvements. It became faster, and role-based access control was added, as well as live tailing of log messages. Of course, all these new features come with a price; the free edition, which I have often recommended for small sites with Cisco logs is completely gone now.
Two years ago LogZilla 5, a complete rewrite became available with many performance improvements under the hood and a new dashboard on the surface. Development never stopped and now LogZilla can parse and enrich log messages and automatically respond to events.
It is an ideal solution for a network operations center (NOC) full of Cisco devices.
Web site: http://logzilla.net/
ELSA – Enterprise Log Search and Archive
Enterprise Log Search and Archive (ELSA) is a centralized logging framework with syslog-ng at its heart. It is the first larger open source project outside of Balabit utilizing the power of the syslog-ng PatternDB log classification tool.
The ELSA architecture is designed to work at a high, continuous incoming message rate, and to bear even higher load peaks for hours. Unlike other solutions that use regular expressions, ELSA utilizes PatternDB, which is more powerful and requires less resources. For example, you can narrow down a search where a given IP address is the target of an operation, rather than perform a general IP address search. ELSA has many features to make the work of sysadmins easier, including a tab-based user interface to run related queries in parallel, scheduled searches, and easy-to-build queries using menus.
In the past few years ELSA improved in many ways. Now it comes with a nice number of bundled message patterns related to security logs, like firewalls, IDS and IPS logs. There are also some plugins, which can enrich logs with additional information, like whois records for IP addresses found in logs, and so on.
For better scalability, the web interface, the database and the indexers can be distributed to multiple machines. This distributed configuration ensures that even an extreme amount of log messages can be searched with Google-like response times.
ELSA is an ideal solution for security teams with large amount of logs, as ELSA can collect and analyze them in near real-time.
The source code is available at https://github.com/mcholste/elsa
Unfortunately ELSA is not actively developed any more. On the other hand there are still many unique features available in ELSA. If you want to give it a try the easiest way is to install Security Onion, which integrates ELSA with a variety of network security software, like Bro, netsniff-ng and more. On the long term ELSA will be replaced by Kibana in Security Onion.
Elasticsearch and Kibana
Elasticsearch is gaining momentum as the ultimate destination for log messages. There are two major reasons for this:
- You can store arbitrary name-value pairs coming from structured logging or message parsing.
- You can use Kibana as a search and visualization interface.
The syslog-ng application can send logs directly into Elasticsearch. We call this an ESK stack (Elasticsearch + syslog-ng + Kibana).
Learn how you can simplify your logging to Elasticsearch by using syslog-ng: https://www.balabit.com/blog/logging-to-elasticsearch-made-simple-with-syslog-ng/
You can also give the Elastic integration of Security Onion a try. Instructions are available at https://github.com/Security-Onion-Solutions/security-onion/wiki/Elastic Note, that this is still Alpha quality at the moment of writing and not recommended for production.
syslog-ng Store Box (SSB)
SSB is a log management appliance built on syslog-ng Premium Edition. SSB adds a powerful indexing engine, authentication and access control, customized reporting capabilities, and an easy-to-use web-based user interface.
Recent versions introduced AWS and Azure cloud support and horizontal scalability using remote logspaces. The new content-based alerting can send an e-mail alert whenever a match between the contents of a log message and a search expression is found.
SSB is really fast, when it comes to indexing and searching log data. To put this scalability in context, the largest SSB appliance stores up to 10 terabytes of uncompressed, raw logs. With SSB’s current indexing performance of 100,000 events per second that equates to approximately 8.6 billion logs per day or 1.7 terabytes of log data per day (calculating with an average event size of 200 bytes). Using compression, a single, large SSB appliance could store approximately one month of log data for an enterprise generating 1.7 terabytes of event data a day. This compares favorably to other solutions that require several nodes for collecting this amount of messages and even more additional nodes for storing them. While storing logs to the cloud is getting popular, on premise log storage is still a lot cheaper for a large amount of logs.
The GUI makes searching logs, configuring and managing the SSB easy. The search interface allows you to use wildcards and Boolean operators to perform complex searches, and drill down on the results. You can gain a quick overview and pinpoint problems fast by generating ad-hoc charts from the distribution of the log messages.
Configuring the SSB is done through the user interface. All of the flexible filtering, classification and routing features in the syslog-ng Open Source and Premium Editions can be configured with it. Access and authentication policies can be set to integrate with Microsoft Active Directory, LDAP and Radius servers. The web interface is accessible through a network interface dedicated to the management traffic. This management interface is also used for backups, sending alerts, and other administrative traffic.
SSB is a ready-to-use appliance, which means that no software installation is necessary. It is easily scalable, because SSB is available both as a virtual machine and as a physical appliance, ranging from entry level servers to multiple-unit behemoths. For mission critical applications you can use SSB in High Availability mode. Enterprise level support for SSB and syslog-ng PE are also available.