Actions

Tags

Peter Czanik
12 Oct 2021
Sending logs to Panther using syslog-ng

Panther is an open-source log management system, which is also available as a service for a time-limited trial. It is still in beta phase, but it looks promising. You can see the “beta” sign on its opening page: https://app.panther.support/ I tested the time-limited cloud service version, but you can also install it locally, either from Dockerhub, or you can build the containers locally from the source.

Even if it is still in beta phase, Panther comes with detailed documentation. There is a notable exception: while syslog-ng is shown on some of the figures, documenting it is still to be done. This blog helps you to get started with sending logs to Panther, using syslog-ng. You can use either legacy syslog with TLS encryption (still a bit problematic) or the http() destination to send logs to the Panther HTTP API.

Before you begin

First of all, you have to set up Panther. Following the documentation at https://openanswers.github.io/panther-docs can help you both if you use Panther in the cloud or if you set it up locally.

While sending logs using the syslog protocol should work even using older versions of syslog-ng, I recommend using a recent version. The reason is simple: configuring TLS is easier when you already have the ca-file() option, which was introduced in syslog-ng version 3.27. If your operating system of choice features an older version, check https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx for up-to-date 3rd-party syslog-ng packages.

Configuring syslog-ng as the Panther syslog source

My tests using legacy syslog with TLS encryption did not work as expected. Some logs arrive in Panther, but most are lost and never show up in the Panther console. Syslog-ng reports all logs as delivered. As you will learn in the Panther API part: the fix might be a template change away...

To configure syslog-ng for Panther, first, you download a small archive from the Panther web interface. Open the Admin tab, and around the middle of the page, you will see the Download button. There is no syslog-ng support yet. Download the rsyslog configuration archive for Linux. It contains a configuration snippet and certificates necessary for TLS encryption. I copied the .pem files into the /etc/syslog-ng/tls.d/ directory. If you use something different, make sure that your syslog-ng configuration is set to that directory.

Then, I created the following syslog-ng configuration. You can save it into the /etc/syslog-ng/conf.d/ directory with a .conf extension, if your distribution of choice supports it. Otherwise append it to syslog-ng.conf.

destination d_panther {
    network(
        "czanik.app.panther.support" port(6514)
        transport("tls")
        tls(
            ca-file("/etc/syslog-ng/tls.d/panther-cert-chain.pem")
            key-file("/etc/syslog-ng/tls.d/key.pem")
            cert-file("/etc/syslog-ng/tls.d/cert.pem")
        )
    );
};

log { source(src); destination(d_panther); };

Note:

  • Change the host name in your configuration.

  • The name of the source in the log statement is most likely something different on your system.

Once you restarted syslog-ng, logs should appear in the Panther web interface.

Configuring syslog-ng for the Panther HTTP API

You can send logs to Panther using its HTTP API. At first, I experienced quite a lot of message loss here as well. After fine-tuning the syslog-ng configuration, it works now rock solid.

The API tab of the Panther web interface shows you how to send logs and also includes examples for Curl. Based on that, I first built a static JSON template. It sometimes worked, but more often not: quotation marks were not escaped. This is why I changed to the format-json template function, as it automatically escapes problematic characters. However, by default, it treats all fields as strings and puts quotation marks around them. Panther expects severity to be a number, so it failed to parse the message. Type hinting resolved the situation. You can find the final configuration below:

destination d_panther_api {
    http(
        url("https://czanik.app.panther.support/api/event/create")
        method("POST")
        headers("X-Api-Token: SOGXXXEUD", "Content-Type: application/json")
        body('$(format-json --pair node=$HOST --pair tag=$PROGRAM --pair summary=$MESSAGE --pair severity=int(1) --rekey * --add-prefix event.)')
    );
};

log { source(src); destination(d_panther_api); destination(d_panther_test);};

Note:

  • Replace the URL in your configuration.

  • Use your own API token, shown both on the API and the Admin tabs in the web interface.

What is next?

Now, that your logs arrive safely to Panther, it is time to explore the rest of the possibilities in the web interface. And you can also enhance the syslog-ng side in many ways. Add a disk-buffer to queue log messages during network problems, use real severity instead of the hard-coded “1”, filter what events you send to Panther, and so on.

-

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.

Related Content