The Elastic Cloud is a service by Elastic providing Elasticsearch and related services in an easy-to-use package. Last year someone reported an issue that it does not work properly with syslog-ng. I did not have time to investigate at that time. Now I started a free trial and soon my log messages from syslog-ng started to appear in Kibana in Elastic Cloud.
From this blog you can learn how to configure syslog-ng for the Elastic Cloud. I go with the most basic settings: exploring Elastic Cloud and syslog-ng Elasticsearch features are both out of scope, as both are very well documented on their respective websites.
Before you begin
On the syslog-ng side, use a recent enough version: I did all my tests using syslog-ng 3.35, the latest available right now, but most likely 3.27 or later should work perfectly.
You also need Elastic Cloud. If you do not have it yet, you can get a free two-week trial account at: https://cloud.elastic.co/registration/ Make sure that you note down the username and password printed on screen right after registration, you will use those to send log messages from syslog-ng.
Configuring syslog-ng
I already mentioned that you need the user-name and password printed during registration (you can change the password later, and most likely also create other users, but I did not explore that area). You need one more piece of information: the endpoint URL for Elasticsearch in your deployment. You can get that when you click on the name of your deployment on the opening page and then on the “copy endpoint” link next to “Elasticsearch” in the list of applications.
Copy the configuration below and either append to your syslog-ng.conf or create a new configuration file with a .conf extension under /etc/syslog-ng/conf.d/, if syslog-ng is configured to use that directory on your system.
destination d_elasticsearch_http {
elasticsearch-http(
index("syslog-ng")
type("")
user("USER")
password("PASSWORD")
url("ENDPOINT/_bulk")
template("$(format-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs
--exclude DATE @timestamp=${ISODATE})")
);
};
log {
source(SOURCE);
destination(d_elasticsearch_http);
flags(flow-control);
};
Replace USER, PASSWORD, ENDPOINT with the username, password and endpoint URL that you collected earlier. SOURCE is a tricky one. It should be replaced with the local log source in your syslog-ng.conf. On openSUSE / SLES, it is usually called “src”, while on Fedora / RHEL “s_sys”.
Testing
Once you reloaded the syslog-ng configuration, log messages start to flow to Elastic Cloud. Open Kibana and add the syslog-ng index. You will find time data in the @timestamp field. If you send logs from a system with systemd / journald, then your log messages will be considerably longer as all field from the journal are also included. Refer to the syslog-ng documentation on how to exclude it from the template.
What is next?
This blog provided you with quick start instructions to syslog-ng and the Elastic Cloud. It should be enough to send your first few log messages for testing. There are a lot more possibilities both in syslog-ng and in the Elastic Cloud. Just think about the template, index name, what you want to send (extra sources, filtering), and so on.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.