The first Logging as a Service (LaaS) I learned about many years ago was created by Loggly. Of course there are many more LaaS providers now. While most services also provide their own clients for sending log messages, many of them also document sending log messages to them using syslog-ng.
From this blog, you can learn about the advantages of using syslog-ng with LaaS, or with any cloud based SIEM solution or a Managed Security Service Provider (MSSP) , and also how easy it is to create a configuration block for a new provider. While creating a configuration block takes some time, the simplification it provides can make your life easier on the long term.
What syslog-ng does
While many people consider syslog-ng to be just another syslog collector, syslog-ng evolved quite a lot over the past two decades. It does not just collect and store syslog messages to files anymore but it can also process and filter messages, and store them to a wide variety of destinations.
Syslog messages are now just one of the many possible log sources in syslog-ng. Next to collecting syslog messages locally or over a network, syslog-ng can read files, sockets, pipes and even application outputs.
“Traditional” syslog implementations sorted log messages based on facility and priority. The syslog-ng application can parse log messages, “look” what is inside and create name-value pairs that can be used for filtering log messages or storing only some specific parts of them. It can also enrich messages, for example, with location information based on IP addresses. Rewriting log messages is useful when you need to mask data, like credit card numbers, due to compliance requirements. All of these make fine-grained log routing possible.
For many years syslog meant saving logs locally to flat files or forwarding logs to a central syslog server and saving the logs there. We have now a lot more possibilities: databases, Big Data, message queuing, storing to different web-based API-s. And of course, sending logs to LaaS providers. Sometimes it works without any extra configuration on the syslog-ng side, sometimes only by adopting a special template. But in either case you can send log messages to LaaS without installing yet another application on your hosts.
Why use syslog-ng with LaaS
Most LaaS has its own client application. Often just a Python or Ruby script, these applications try to follow text files on the system and upload them to the LaaS provider. While they do their jobs most of the time, syslog-ng is a good replacement in most cases. Here I list a few reasons, why:
-
Installing a 3rd party application, even if as simple as a logging agent, takes significant effort to push through security and operation teams. The syslog-ng application is often already part of the Linux distribution and thus does not need a separate clearance.
-
Often there is more than one LaaS or analytics software to feed with log messages. You can route log messages from syslog-ng to all of them instead of installing a dedicated application for each of them.
-
The syslog-ng application is often used to create a central archive of your log messages. You can keep using it this way. Process and filter log messages at your local syslog-ng server and then save a larger subset of log messages on the central server and forward what is really necessary to LaaS. With this, you can save on the direct LaaS charges, and also keep the required network bandwidth at a lower level. Having a local archive helps not just when you have to find some extra information about events but long-term log retention is usually cheaper locally than in an LaaS.
-
Before sending logs to LaaS, you can process your logs in syslog-ng. You can enrich messages, like adding the role of a machine. Or you can mask sensitive information, like replacing usernames with hashes. If the cloud service finds a suspicious event in your logs, you can still check the locally archived logs to find the real username.
How to use LaaS from syslog-ng
Most LaaS providers document how to send logs to their services through syslog-ng. In some cases, it is just setting a specific host and port name in the configuration. In the majority of cases it means setting up a specific template in syslog-ng with one or more user specific unique identifiers. Most LaaS now also features an HTTP-based API to collect log messages. While providers do not document how to access these APIs from syslog-ng, with a bit of research it is possible to use them as well. Using the http() destination has some additional benefits, like using a proxy, often more flexibility on the payload side, built-in load-balancer and more.
When all you have to do is to use a specific destination address in your configuration, there is not much you can simplify on the process. But once you have to use templates, you can greatly simplify how you refer to the given LaaS destination.
You can take a look at the documentation of loggly() destination of syslog-ng. Once you have an LaaS directly supported within syslog-ng, yall you have to do is specify a unique ID or token in the configuration.
Right now Loggly and SumoLogic are supported by syslog-ng, and most likely also DataDogHQ (syslog-ng supports LogMatic, which was bought recently by DataDogHQ). Support here means that a configuration snippet in the syslog-ng configuration library (SCL) provides an easy to use destination and hides away the details from the users.
How to support a new LaaS
For a service not yet supported within syslog-ng, you can create your own SCL and submit it to syslog-ng developers at https://github.com/syslog-ng/syslog-ng/. It means not only sharing your solution with the community but you can receive reviews and helpful suggestions by syslog-ng developers and fellow users.
Supporting a new LaaS is not rocket science. Take a look at the Loggly syslog-ng documentation and then check the related configuration snippet in the SCL. The Loggly and other configurations are available at https://github.com/syslog-ng/syslog-ng/tree/master/scl If you want to use a HTTP API, check the SumoLogic configuration or take a look at the elasticsearch-http() destination. Just copy the configuration file to /etc/syslog-ng/conf.d/ rename the block and do the necessary changes for the given LaaS destination. For example, I do not have a LogDNA account, but based on the documentation it seems that all it needs is just a change of block name and host name compared to Loggly.
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.