Actions

Tags

Peter Czanik
29 Nov 2023

Recently, syslog-ng 4.5.0 was released with many new features. These include sending logs to OpenObserve using its JSON API, support for Google Pub/Sub, a new macro describing message transport mechanisms like RFC 3164 + TCP, an SSL option to ignore validity periods, and many more. You can find a full list of new features and bug fixes in the release notes at: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.5.0

In this blog, you can find some pointers on how to install the very latest syslog-ng version and learn how you can configure syslog-ng to use the OpenObserver JSON API.

Getting syslog-ng 4.5.0

There is a new syslog-ng version almost every second month. Most Linux distributions cannot keep up with this speed, as they have a new release only every half a year (or just every 3-5 years). Right now, even the Repology website is lagging behind, as they consider 4.4.0 to be the latest version at the time of writing this blog: https://repology.org/project/syslog-ng/versions

Repology lists only official distro repositories or semi-official repos, like RHEL EPEL. There are also some unofficial syslog-ng repositories for major Linux distributions, which you can use to install the very latest syslog-ng version, often with more features than what is available in Linux distributions. For more information check the related blogs:

After a bit of delay, I usually make syslog-ng available in FreeBSD ports and in my syslog-ng-stable RPM repositories as well. This time, however, I spotted a number of smaller bugs after the release became available. Because of this, my safety delay might be longer this time, or I might even skip 4.5.0 altogether.

You can read about why some of the features are not available everywhere from one of my earlier blogs: https://www.syslog-ng.com/community/b/blog/posts/why-is-a-feature-not-available-in-the-syslog-ng-package

Using the OpenObserve JSON API

I covered OpenObserve in this blog just a couple months ago. At that time, I showed you how to ingest logs to OpenObserve using the Elasticsearch bulk API: https://www.syslog-ng.com/community/b/blog/posts/sending-logs-to-openobserve-using-syslog-ng However, I have heard about OpenObserve even more often from users ever since. Then, version 4.5.0 introduced official support for the OpenObserve JSON API. Working with OpenObserve this way is not much different but is slightly easier to configure.

Configuring syslog-ng was already easy, as OpenObserve provides the user with a copy paste-ready syslog-ng configuration snippet in its user interface under the “ingest” menu. You only need to make sure that you use the right log source – otherwise, the configuration is ready to use.

The configuration below shows you both the destination configuration generated by OpenObserve and the version using the new openobserve-log() destination, which utilizes the OpenObserve JSON API instead of the Elasticsearch API.

destination d_openobserve_http {
    elasticsearch-http(
        index("syslog-ng")
        type("")
        user("peter@czanik.hu")
        password("0PsCZD4qCBp6UVyT")
        url("http://172.16.167.180:5080/api/default/_bulk")
        template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE --key ISODATE @timestamp=${ISODATE})")
    );
};

destination d_openobserve_jsonapi {
    openobserve-log(
        user("peter@czanik.hu")
        password("0PsCZD4qCBp6UVyT")
        url("http://172.16.167.180")
        stream("syslog-ng")
        port(5080)
    );
};

source s_tcp {
  tcp(port(514));
};


log {
    source(s_tcp);
    destination(d_openobserve_jsonapi);
    flags(flow-control);
};

The d_openobserve_http destination is what OpenObserve automatically generates. It uses the Elasticsearch API. It is only there for reference, as it is not utilized in this configuration in a log path.

The d_openobserve_jsonapi destination uses the OpenObserve JSON API to forward log messages. The two destinations are almost equivalent. The only difference is that the destination using the JSON API does not have any templates configured. For the openobserve-log() destination, it is called record() instead of template(), as it is not a full template: it contains only the name-value pair selectors for the format-json template function.

The configuration above also adds a tcp() source and a log path, which connects this tcp() source to the d_openobserve_jsonapi destination.

Testing

Testing is easy: just send some logs to port 514 on the host. If you are on localhost, then something similar should be OK (but note that logger parameters might be different on your host):

logger --rfc3164 -T -n 127.0.0.1 -P 514 this is a test massage1

Right after you send this message, it should show up in your next query in the syslog-ng stream in the OpenObserve search interface.

What is next

The new, dedicated OpenObserve destination is just one of the new features of syslog-ng 4.5.0. I hope that this overview made you interested in installing syslog-ng 4.5.0 on a test system and check out other new features as well.

-

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.

Related Content