Actions

Tags

Peter Czanik
21 Nov 2023

Logging into Humio (which was recently re-branded to Falcon LogScale) was available for years, using their Elasticsearch compatible API. However, according to Humio developers, it is slightly slower than other APIs for log ingestion. Axoflow contributed a Logscale destination to syslog-ng, which uses Logscale’s native API. I did not measure if there is really a performance difference, however it is definitely easier to configure it.

Before you begin

The logscale() destination was introduced in syslog-ng version 4.3.1. If your operating system has an older version of syslog-ng, check our 3rd party repositories page, if there is something more recent for your OS: https://syslog-ng.org/3rd-party-binaries/.

You also need a Humio / Logscale account, a token and a URL. The default URL in the destination is suitable for commercial customers outside of the EU. There is a different URL for customers in the rest of the World. Community users have yet another URL, I will use that one in my examples.

The old way

You can read more about how to ingest logs with Humio using the Elasticsearch API at https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-humio-using-the-elasticsearch-http-destination-of-syslog-ng. Here I just want to copy & paste the configuration for reference:

destination d_elastic_humio {
    elasticsearch-http(
        type("humio") # not used by humio, but required by plugin
        index("syslog-humio") # not used by humio, but required by plugin
        url("https://cloud.community.humio.com/api/v1/ingest/elastic-bulk")
        workers(2)
        batch-lines(200)
        user("syslog-ng") # not used by humio, can be whatever you want
        password("a555ef1c-XXX-YYY-ZZZ-63e6914e22e1")
        template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --exclude .journald.*
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE  @timestamp=${ISODATE})")
    );
};

The mandatory parameters are type(), index(), url(), user() and password(), the others just fine-tune how the destination works.

Configuring the logscale() destination

If you are a non-EU commercial customer, your job is very easy: the only mandatory parameter is the token:

destination d_logscale {
  logscale(
    token("my-token")
  );
};

My configuration is slightly more complex, as I also configured the url() parameter:

destination d_logscale {
  logscale(
    token("a5xxxx1c-xxxx-42b1-xxxx-63e6xxxx22e1")
    url("https://cloud.community.humio.com")
  );
};

What is next?

If you take a look at the GitHub pull request, you will see a few more available configuration parameters: https://github.com/syslog-ng/syslog-ng/pull/4472. And if you take a closer look at the SCL itself, which implements the logscape() destination, you will see that you can change the attributes() parameter to change the content of log messages. You can view it on-line at https://github.com/syslog-ng/syslog-ng/blob/master/scl/logscale/logscale.conf.

-

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.

Related Content