General availability of Elasticsearch 8 was announced last week. There were quite a few rumors that it will break compatibility with third party tools. I tested it as soon as I had a little time: I am happy to share that anything I tested with the elasticsearch-http() destination of syslog-ng still seems to work perfectly well with the latest version of Elasticsearch.
Before you begin
If you want to try Elasticsearch 8, installers for many platforms are available on the Elastic website: https://www.elastic.co/downloads/elasticsearch I installed it on Rocky Linux 8 together with the latest syslog-ng, version 3.35. Earlier versions of syslog-ng should work fine, starting from syslog-ng version 3.23.
The good
For me, who uses Elasticsearch only in non-production environments, enforcing security is slightly annoying. However, it is a fantastic step, especially knowing the number of breaches due to security being a paying feature for many years at Elastic. Even when it became freely available, it was not enabled by default and was difficult to configure. In version 8.0, TLS and password protection are both enabled by default and are easy to configure. When RPM installation finished, instructions for next steps were printed on screen.
Configuring syslog-ng
Most syslog-ng configuration examples use plain HTTP and no password protection. Luckily, enabling these is easy in the syslog-ng configuration.
destination d_elasticsearch_http {
elasticsearch-http(
index("syslog-ng")
type("")
user("elastic")
password("Gr3Cmh7-5CdZYGl+uCWB")
url("https://localhost:9200/_bulk")
template("$(format-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs
--exclude DATE @timestamp=${ISODATE})")
tls(peer-verify(no))
);
};
There are four notable lines in the above configuration:
-
user() sets the user name you use to send logs. Here I used the system user, not recommended for production use.
-
The password() sets the password of the above user.
-
The url() is configured to use https instead of http.
-
Verifying the peer is disabled. Not recommended for a production environment, but good enough for a quick test.
These are all security-related settings, everything else stayed the same.
The bad
For the past half decade I have always tested Elasticsearch on virtual machines with 4 GB of RAM and 2 GB of swap. Today, once I finished setting up Elasticsearch I configured syslog-ng and wanted to send some logs. I got a “connection refused” message. It turned out that both Elasticsearch and Kibana were killed by the OOM Killer for using too much memory. Everything else was running nicely on the system.
What is next?
So far, I’ve only done some very quick functionality tests to check if the elasticsearch-http() destination of syslog-ng still works and everything seems to work perfectly, just like before. I haven’t done any performance tests yet, but the whole thing feels to be slower. In the coming weeks I’m planning to check Elasticsearch 8 in more detail to see if the additional resource usage is really worth it.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @Pczanik.