Each time a new major Elasticsearch version is released, someone asks if it works with syslog-ng. So I gave it a quick test and based on that, it works fine. But of course, some terms and conditions apply… :-)
Before you begin
On the syslog-ng side, I used the latest development snapshots for RHEL: https://www.syslog-ng.com/community/b/blog/posts/rpm-packages-from-syslog-ng-git-head For Elasticsearch, I used a tutorial from the Elastic website: https://www.elastic.co/guide/en/elastic-stack/9.0/installing-stack-demo-self.html
Make sure that you use a dedicated test environment. Why? See below… :-)
Operating systems
For most software, when their documentations mention RHEL 8 as a system requirement, said software usually work without any problem on RHEL 9 and even on CentOS Stream 10 as well. However, this does not seem to be the case with Elasticsearch: its tutorial says that “the examples in this guide use RPM packages to install the Elastic Stack components on hosts running Red Hat Enterprise Linux 8”. My initial test was done on RHEL 9, however, and I could not get Elasticsearch 9.0.0-beta1 to work – even the curl test failed. I wanted to check if the issue was a software or an environment problem, so I installed the latest Elasticsearch 8.X release in the VM. But I got the same results – I could not get it to work.
So I went back to RHEL 8 and suddenly, installing Elasticsearch 8 worked perfectly. The curl test worked, just as installing Kibana. I sent logs from syslog-ng and I could browse the results in Kibana immediately.
Fresh install instead of upgrading
I am lazy, so once I had a working 8.X installation, I tried to upgrade it to version 9.0.0-beta1. But I had no luck. Deep in the log file among some long Java error messages, there was a short message that upgrading from 8.X to 9.0 is not supported. So I deleted the Elasticsearch and Kibana packages, along with all the configuration and data directories. I installed Elasticsearch 9.0.0-beta1 again from scratch, then suddenly, everything worked as expected. I have yet to make some measurements, but at first glance, version 9 feels to be faster.
What is next?
If you use RHEL or compatibles and still have an old RHEL 8 (or compatible) system around, you are ready for testing. Everything worked fine in my test environment, but as its name implies, I did not have production logs available. If you have a chance, set up a RHEL 8 (or compatible) box, prepare a fresh Elasticsearch & Kibana 9.0.0 beta installation, then feed it with production logs. Just make sure that your test destination is set up alongside a well-working production destination instead of replacing it. Let us know your experiences!
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.