Actions

Tags

Peter Czanik
27 Aug 2024
First steps with Quickwit and syslog-ng

We are always looking for new ways to store log messages. Quickwit is a new contender, designed for log storage, and among others, it also provides an Elasticsearch-compatible API.

From this blog, you can learn about Quickwit, and how to forward log messages from syslog-ng to it using the Elasticsearch-compatible API.

Before you begin

On the syslog-ng sid,e you can use any version supported by your operating system. A few months ago, I would still mention that you need at least version 3.23, but with RHEL 7 end of life, version 3.23 is the earliest version still actively used.

All you have to make sure on the syslog-ng side is that you install http protocol support. It comes enabled by default in the FreeBSD package, and in a few Linux distributions. In other packages and distributions, you have to install a sub-package for it. The name varies among distros: syslog-ng-curl, syslog-ng-http, or syslog-ng-mod-http.

You also need Quickwit. While syslog-ng is available on many platforms (including Linux, FreeBSD, MacOS, and various 32bit CPUs), Quickwit needs Linux on 64bit x86. Anything else is just experimental. I used the simplest possible way for installation: a container. You can also install Quickwit directly on your OS or in Kubernetes, but both alternatives are more difficult than using containers.

Starting Quickwit

Quickwit is a search engine, just like Elasticsearch, Opensearch and many others. Its web interface is very basic, as you are supposed to send logs and search logs via an API. Depending on your use case, sharing a single port from the container is all you need to get started.

By the time I am writing this blog, the version number still did not reach 1.0, so expect some rough edges. As far as I can see, there is no TLS for network connections, no authentication, and so on. Still, it works, it is fast, and it looks promising. According to the Quickwit website, they already have users with petabytes of data.

Note that the command line examples show Docker, but I used Podman, and it worked perfectly well as well.

First, create a data directory:

mkdir qwdata

It is not strictly necessary, but you can “download” the image in a separate step, and print version information:

docker run --rm quickwit/quickwit –version

The following command starts Quickwit using the previously created data directory and exposing the REST API port on the localhost:

docker run --rm -v $(pwd)/qwdata:/quickwit/qwdata -p 127.0.0.1:7280:7280 quickwit/quickwit run

As I mentioned earlier, Quickwit has no security included yet, so it is better to bind it to localhost, even in a test environment. Of course, you can also add some security yourself, but that is not the scope of a quick test.

Creating and index

Before you can store logs to Quickwit, you have to create an index. For that, you need a yaml file describing the index, and uploading it to Quickwit.

version: 0.7

index_id: syslog-ng

doc_mapping:
  field_mappings:
    - name: timestamp
      type: datetime
      fast: true
      input_formats:
        - rfc3339
  timestamp_field: timestamp

This was the simplest file I could create based on the examples I found in the documentation. I saved it with the name: index.yaml

Use curl to upload the yaml file to the API:

curl -XPOST http://127.0.0.1:7280/api/v1/indexes --header "content-type: application/yaml" --data-binary @./index.yaml

Quickwit is now ready to receive your log messages.

Configuring syslog-ng

On the syslog-ng side, you have to configure an elasticsearch-http() destination. There is no encryption or authentication, so the configuration is really simple. You can append the following snippet to the end of your syslog-ng.conf, or create a new file under the /etc/syslog-ng/conf.d/ directory if the syslog-ng package in your OS is configured to use it:

destination d_elasticsearch_http {
    elasticsearch-http(
        index("syslog-ng")
        type("")
        url("http://localhost:7280/api/v1/_elastic/_bulk")
        template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE timestamp=${ISODATE})")
    );
};
log {
  source(src);
  destination(d_elasticsearch_http);
};

The name of the source might be different in your environment. Here it is the source for local logs on an openSUSE system.

The important parameters in the elasticsearch-http() destination are:

  • index: the name of the index where syslog-ng stores the log messages

  • url: the URL where syslog-ng connects. Note that there is no encryption, and the port number and path are specific to Quickwit.

  • template: which name-value pairs you want to forward to Quickwit. Make sure that the name for the time stamp is the same here as in the yaml file used to create the index.

Testing

Once you saved your syslog-ng configuration and reloaded syslog-ng, you should see logs arriving to Quickwit. If there are no logs, you can use the “logger” utility to create some log messages. I used the Quickwit API from a web browser to browse logs:

What is next?

Quickwit is not at version 1.0 yet, but already used in production. Of course, my blog is enough to make the first steps with Quickwit and syslog-ng, but far from enough for production use. You have to implement some kind of security yourself. You also need to fine-tune index creation, as my example is just the absolute minimum.

-

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.

Related Content