Support for Elasticsearch was updated recently in both the Open Source (version 3.7.2) and the Premium Edition (version 5 F5) of syslog-ng. Changes were the same for both editions and brought more speed and simplicity to the Elasticsearch driver.
This is the first blog post in a six-part series on storing logs in Elasticsearch using syslog-ng. You’ll find a link to the next part in the series at the end of this post. You can also read the whole Elasticsearch series in a single white paper.
Configuring Elasticsearch destination
Let’s start with simplicity. As you might know, while most of syslog-ng is developed in C, the Elasticsearch destination is implemented in Java. This enables the use of the official Elasticsearch client JAR files. On the other hand, the configuration syntax of Java-based destination drivers was a bit difficult, as it added an extra layer and parameters. This is now well hidden by using the SCL (syslog-ng configuration library) for configuring the Elasticsearch destination. Your configuration can be as simple as this:
elasticsearch( index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("test") cluster("syslog-ng") );
Of course, all the parameters are still configurable, but there are reasonable defaults for most of them.
This release also enabled the possibility to send concurrent requests to Elasticsearch servers with the concurrent-requests() option, which can increase performance considerably. Defaults were also changed to provide better out-of-the-box performance. Note that there is trade-off between batch size, message throughput and risk of losing messages in case of failure.
Send logs directly to Elasticsearch
This November I have visited four conferences either as an exhibitor or as a presenter. During these conferences I had the opportunity to talk to hundreds of syslog-ng users. Many of them mentioned that alongside syslog-ng they also use Logstash to pump log messages into Elasticsearch. They were very happy to learn that syslog-ng can now send logs directly to Elasticsearch destinations. The use of syslog-ng can greatly simplify the logging architecture, and also speed it up while reducing resource usage considerably.
As I have mentioned earlier, most of syslog-ng is implemented using the C programming language. This makes log collection and processing extremely efficient. Java code is only used in the final step, when logs are sent off to Elasticsearch.
Collect system- and application logs
Logstash, or its dedicated file reader component, Filebeat, requires syslog to be installed on machines to be able to collect system logs. In comparison, syslog-ng can be used to collect both system and application logs. There is no need for a separate file reader or a separate system log collector. The syslog-ng application can also perform all the log processing-related tasks, which means that a single application can be used instead of having to configure three different software (syslog + filebeat + Logstash) for the same tasks.
As you can see, syslog-ng can provide you with a simple and efficient architecture for collecting and processing log messages.
Check https://www.syslog-ng.com/community/b/blog/posts/getting-started-with-syslog-ng-3-7-1-and-elasticsearch-hadoop-kafka for syslog-ng OSE package availability and getting started.
Request a trial version of syslog-ng PE at https://www.syslog-ng.com/register/115488/ if you need commercial support!
In my next Elasticsearch blog, I cover further details on the basics of how you can collect logs using syslog-ng and send them to Elasticsearch for storing.
Read the entire series about storing logs in Elasticsearch using syslog-ng in a single white paper.