Opensearch is a fork of the Elastic stack code base, made right before the license change. The first release candidate (RC1) has been released recently. Next to plain text files, Elasticsearch is one of the most popular destinations in syslog-ng, but after the license change people started to look for alternatives. I did some quick tests and using the elasticsearch-http() destination, syslog-ng seems to work fine with Opensearch as well.
Opensearch is not yet production ready. It is still in testing phase. However, if the licensing changes of Elastic makes you search for alternatives, switching to Opensearch might be the easiest. RC1 is already in a good enough shape to start testing it, so you can switch easier once it is ready for production.
You can learn from this blog how to get started with Opensearch, dashboards and syslog-ng. Another alternative that syslog-ng users explored is Grafana Loki.
Disclaimer: covering a given technology or brand on the syslog-ng blog is not an endorsement. The syslog-ng blog covers new syslog-ng features, new trends in log management or questions, problems coming up in the syslog-ng community.
Before you begin
There are no RPM or DEB packages of Opensearch available yet. You can either download binaries as compressed TAR archives, or use Docker. The use of Docker compose can greatly simplify testing, as it sets up a ready-to-use, local, two-node cluster together with Dashboards (the name of the Kibana fork) in just a few minutes.
I did my tests on openSUSE and Docker, but any operating system with a recent enough Docker version and docker-compose should work. Not tested, but a recent-enough podman and podman-compose should work as well.
On the syslog-ng side, anything after syslog-ng 3.21 should work, but a more recent version is recommended. If your OS features an earlier version of syslog-ng, check https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx if up-to-date syslog-ng is available from 3rd-party repositories. You need http() support enabled. On FreeBSD, it is enabled by default. On Fedora/RHEL you also need the syslog-ng-http sub-package, on openSUSE syslog-ng-curl.
Installing the Opensearch stack
Getting started with Opensearch is really easy when using the docker-compose method. Download docker-compose.yml from the Opensearch website to an empty directory. From this point, it is a single command:
docker-compose up
In a few minutes, you can already point your browser at port 5601 of the host where you installed the Opensearch stack. Your first surprise comes when you try to reach the page: it is password protected. The default user/password pair is admin/admin, which you should change unless everything is running on your laptop behind a firewall.
Look around. The interface will look familiar, still a bit strange. It is a lot cleaner than the regular Kibana web interface and thus easier to find anything that you need.
Configuring syslog-ng
The next surprise comes when you try to send logs to Opensearch using syslog-ng. When you try the following destination, sending logs to Opensearch will fail:
destination d_opensearch {
elasticsearch-http(
index("syslog-ng")
url("https://localhost:9200/_bulk")
type("")
};
Why? It works with out-of-the-box Elasticsearch, which has no security enabled. Opensearch comes with security enabled, outof-the-box. The port is TLS-encrypted and requires a valid user and password. For a test, you can use the default admin/admin, but of course it is not recommended for production use.
The TLS part is a bit tricky. You have to extract root-ca.pem from the Opensearch container and make it available under the /etc/syslog-ng/ directory structure. I created a new sub-directory called “tls”, changed to it, and then used the “docker cp” command to extract the file from the running container. The container ID will be different on your system, use “docker ps” to figure it out. Then:
docker cp 72b9b293941f:/usr/share/opensearch/config/root-ca.pem .
This command extracts root-ca.pem from the Opensearch container and copies it into the current directory. You do not need this step when you use your own PKI in a production environment. But here we use the demo keys provided in the container image.
You can now enable password and TLS support in syslog-ng. The following config snippet sends all local log messages to Opensearch on an openSUSE host:
destination d_opensearch {
elasticsearch-http(
index("syslog-ng")
url("https://localhost:9200/_bulk")
type("")
user("admin")
password("admin")
ca-file("/etc/syslog-ng/tls/root-ca.pem")
);
};
log {
source(src);
destination(d_opensearch);
};
You might need to use a different source name on your system.
Testing
Once you reloaded syslog-ng and the configuration took effect, you should be able to see incoming log messages in Opensearch. First you need to go to “Stack Management” and configure an index pattern. “syslog-ng” should already be available there. Then you can go to “Discover” and view the incoming log messages.
What is next?
From this blog, you could learn how to quickly set up a set environment for Opensearch and syslog-ng. Now comes the hard part: reading the documentation, learning how to configure individual components, rolling out your own PKI, and more. Those topics are well beyond the scope of a syslog-ng blog.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @Pczanik.
