Actions

Tags

Peter Czanik
6 Sep 2023

A question was asked if syslog-ng can send logs to OpenObserve. It has an Elasticsearch compatible API for log ingestion, but syslog-ng is not mentioned in the documentation. My plan was to document how to modify the syslog-ng elasticsearch-http() destination, based on API documentation. However, as it turned out, OpenObserve has a ready to use syslog-ng configuration example in the web UI.

Before you begin

So, what is OpenObserve? It is an open-source observability platform, which can store and monitor logs, metrics and traces. It provides an Elasticsearch compatible API for log ingestion. OpenObserve is available in two forms: as a cloud service and as a self-hosted installation. In my tests I used the self-hosted version. There are instructions for Mac, Windows, and Linux, and for containers. Using the binary did not work in my environment (Rocky Linux 8), so I went for the container version. Ingestion does not seem to have TLS encryption, which is good enough for testing. However, make sure that the OpenObserve API is well protected in a production environment.

You can learn more about OpenObserve on their website: https://openobserve.ai/ If you want to go the self-hosted route, check https://openobserve.ai/docs/quickstart/ for installation instructions.

On the syslog-ng side you need at least version 3.23 with JSON and HTTP support enabled. All these features are built into the FreeBSD package, but in most Linux distributions HTTP support is available only as a separate sub-package.

Configuring syslog-ng

When you first login to OpenObserve, the opening page has a link, which brings you to ingestion. Once you have data, this link is not shown, but you can still find the very same page from the left-hand icon menu. The ingestion menu has three sub-menus: logs, metrics and traces. From those we need logs. My original plan was to configure the elasticsearch-http() destination of syslog-ng based on how to use curl. This was not necessary: there is a ready-to-use syslog-ng configuration snippet as well.

When you click on “syslog-ng” in the menu, you will see something similar:

destination d_openobserve_http {
    elasticsearch-http(
        index("syslog-ng")
        type("")
        user("peter@czanik.hu")
        password("dQhxLeWJRwg4qwhm")
        url("http://172.16.167.180:5080/api/default/_bulk")
        template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE --key ISODATE @timestamp=${ISODATE})")
    );
};


log {
    source(src);
    destination(d_openobserve_http);
    flags(flow-control);
};

As you can see, it is not a complete configuration. It is rather something to drop into a file under /etc/syslog-ng/conf.d/ or append to your syslog-ng.conf. There is one thing you must change, or at least double check: the name of the source. In quite a few distros the local log source is called “src”, just as in the configuration snippet in the UI. However, in my syslog-ng.conf the local log source is called “s_sys”.

Note: if you use Docker and collect local logs, you might end up with an endless loop: Docker logging to syslog-ng, syslog-ng sending logs to OpenObserve, OpenObserve sending logs to Docker…

So, I ended up using a network source to feed OpenObserve:

source s_tcp {
  tcp(port(514));
};

destination d_openobserve_http {
    elasticsearch-http(
        index("syslog-ng")
        type("")
        user("peter@czanik.hu")
        password("dQhxLeWJRwg4qwhm")
        url("http://172.16.167.180:5080/api/default/_bulk")
        template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE --key ISODATE @timestamp=${ISODATE})")
    );
};


log {
    source(s_tcp);
    destination(d_openobserve_http);
    flags(flow-control);
};

The parameter of index() is called “stream” within OpenObserver. The user() parameter is the same as the e-mail address you use to login to the web interface. However, the password() parameter is a token, different from your login password. It is random and can be regenerated. The template() is perfect in most situations, but you might want to fine tune if you only want a subset of your name-value pairs included.

Testing

If you use the configuration above, you can send some test messages using logger:

logger -T -P 514 --rfc3164 -n 127.0.0.1 this is a test

You should see the test messages when you click on “Logs” in the right-hand menu.

What is next?

OpenObserve looks like a promising, light weight alternative to Elasticsearch / OpenSearch. However, the version I tested is 0.5.2, so it is not even 1.0. It is worth putting on your radar, even install it and feed it with data which is not mission critical. I am looking forward to seeing a version beyond 1.0.

-

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.

Related Content