Last time we looked at how syslog-ng can send logs to Quickwit using its Elasticsearch compatible API. This time we are going to look at how to use the OpenTelemetry protocol to send logs to Quickwit with syslog-ng.
Before you begin
On the syslog-ng side you need at least version 4.3 with OpenTelemetry support enabled, but the newer the better. Note that OpenTelemetry support does not seem to compile on 32-bit systems or anything older. Support for FreeBSD, MacOS and EPEL 9 was enabled in syslog-ng version 4.8. For the rest, it is mostly rolling Linux distributions that support it. I used openSUSE Tumbleweed while writing this blog. Support for OpenTelemetry is usually available as a separate sub-package, called syslog-ng-opentelemetry, or something similar. On FreeBSD you need to compile syslog-ng yourself from ports as this feature is not enabled by default.
You also need Quickwit. While syslog-ng is available on many platforms (including Linux, FreeBSD, MacOS, and various 32-bit CPUs), Quickwit needs Linux on 64-bit x86. Anything else is just experimental. I used the simplest possible way for installation: a container. You can also install Quickwit directly on your OS or in Kubernetes, but both alternatives are more difficult than using containers.
Starting Quickwit
Quickwit is a search engine, just like Elasticsearch, Opensearch and many others. Its web interface is very basic, as you are supposed to send logs and search logs via one of the APIs. As of writing this blog, Quickwit is still not at version 1.0, so expect some rough edges. As far as I can see, there is no TLS for network connections, no authentication, and so on. Still, it works, it is fast, and it looks promising. According to the Quickwit website, they already have users with petabytes of data.
Note that the command line examples show Docker, but I used Podman, and it worked just as well.
First, create a data directory:
mkdir qwdata
It is not strictly necessary, but you can “download” the image in a separate step, and print version information:
docker run --rm quickwit/quickwit –version
The following command starts Quickwit using the previously created data directory and exposing two ports. The first one is the REST API, also providing a very simple web interface. The second one is for OpenTelemetry:
docker run --rm -v $(pwd)/qwdata:/quickwit/qwdata -p 7280:7280 -p 7281:7281 quickwit/quickwit run
As I mentioned earlier, Quickwit has no security features yet, so do not use the above setting in a production environment. Of course, you can also add some security yourself, but that is outside the scope of this blog.
Configuring syslog-ng
On the syslog-ng side, you have to configure an opentelemetry() destination. There is no encryption or authentication, so the configuration is really simple. You can append the following snippet to the end of your syslog-ng.conf, or create a new file under the /etc/syslog-ng/conf.d/ directory if the syslog-ng package in your OS is configured to use it:
destination d_qw_otel {
opentelemetry(
url("172.16.167.183:7281")
);
};
log {
source(src);
destination(d_qw_otel);
};
The name of the source might be different in your environment. Here it is the source for local logs on an openSUSE system.
Testing
Once you saved your syslog-ng configuration and reloaded syslog-ng, you should see logs arriving to Quickwit. Logs received using the OpenTelmetry protocol are stored in the index called “otel-logs-v0_7”. If there are no logs, you can use the “logger” utility to create some log messages. I used the Quickwit API from a web browser to browse logs:
What is next?
As you can see, sending logs from syslog-ng to Quickwit using the OpenTelemetry protocol works. However, if you check back to my previous blog, where we used the Elasticsearch-compatible API of Quickwit for sending logs, you will see that there are a lot more details preserved. Unless you are happy with just the message and date, without any of the additional information, using the Elasticsearch-compatible API is a more practical choice.
Quickwit is still not at version 1.0, however, it has already been deployed in production environments in some cases. Of course, my blog is only enough to make the first steps with Quickwit and syslog-ng, but far from enough for production use. You have to implement some kind of security yourself.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.