Sending logs from syslog-ng to Grafana Loki

Loki is one of the latest applications that lets you aggregate and query log messages, and of course to visualize logs using Grafana. It does not index the contents of log messages, only the labels associated with logs. This way, processing and storing log messages requires less resources, making Loki more cost-effective. Promtail, the log collector component of Loki, can collect log messages using the new, RFC5424 syslog protocol. This is where syslog-ng can send its log messages.

From this blog, you can learn a minimal Loki & Promtail setup. We will send logs from syslog-ng, and as a first step, will check them with logcli, a command line utility for Loki. Once it works, we will also install Grafana in a container and query Loki from there.

Before you begin

All my tests were done on a CentOS 7 virtual machine. Loki, Promtail and Grafana open a number of ports, and syslog-ng needs to send logs to a non-standard port. To make my life easier, I put SELinux into permissive mode and disabled the firewall (of course neither of these are recommended in a production environment). On openSUSE and Ubuntu, you also need to open ports for the firewall, but instead of SELinux, you need to deal with AppArmor. CentOS 8 has podman instead of Docker. It most likely works, just replace docker with podman on the command line.

I followed instructions from the relevant Loki documentation, but it failed on me a number of times. So instead of pointing you to the documentation, I rather give you quick instructions here together with the necessary workarounds. That said, if you need it, you can find the Loki documentation at https://grafana.com/docs/loki/latest/

On the syslog-ng side, any version supporting RFC5424 syslog can be used. Any syslog-ng version from the last decade will work. The one in CentOS 7 EPEL works perfectly, but I used the latest unofficial rpm package.

As with most of my blogs, remember that the setup I show you below is good enough for getting to know the functionality of the software, but a production setup needs a lot more planning and configuration. The sample configurations store files in /tmp and in the container. There is no encryption configured, nor any other features that are otherwise needed in a production environment.

Downloading Loki

Loki, Promtail and logcli can be downloaded from GitHub. I used the latest version, that is 1.6.1. All of these are single binaries without additional dependencies. For simplicity’s sake, create a fresh new directory and download the applications there. Open https://github.com/grafana/loki/releases/ in a browser and search for assets. Download the linux-amd64 variant of Loki, Promtail and logcli. These are zip files, so you also need to extract them.

Configuring Loki

There is an initial configuration available for download both for Loki and Promtail. Download them to the same directory:

wget https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/master/cmd/promtail/promtail-local-config.yaml

Now try to start Loki:

./loki-linux-amd64 -config.file=loki-local-config.yaml

If you get an error message related to “ruler”, remove the last section of the configuration and start Loki again. This option will work only in an upcoming version of Loki.

Before starting Promtail, change the last section of the file to collect log messages using the syslog protocol:

scrape_configs:
  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:1514
      idle_timeout: 60s
      label_structured_data: yes
      labels:
        job: "syslog"
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'

And now you can start Promtail as well:

./promtail-linux-amd64 -config.file=promtail-local-config.yaml

Configuring syslog-ng

The configuration below shows you how to send log messages from the same host to the open Promtail port. If you send logs from a remote host, change “localhost” to the external IP address of the host running Promtail.

destination d_loki {
  syslog("localhost" transport("tcp") port(1514));
};

log {
  source(s_sys);
  destination(d_loki);
};

Note that the source name for locally collected log messages might be different on your system.

Once syslog-ng is reloaded, it will send all locally collected log messages to Promtail using the RFC5424 syslog protocol.

Testing Loki using logcli

To keep the logcli command line a bit shorter, you can define the address of Loki as an environment variable:

export LOKI_ADDR=http://localhost:3100

Otherwise you can specify it using the --addr option on the command line. The easiest way to get started is tailing the logs, similarly to regular syslog messages (tail -f /var/log/messages):

./logcli-linux-amd64 query --tail '{job="syslog"}'

This command line means that logcli shows any new logs with the “syslog” label. You should see log messages from syslog-ng on screen, similar to this:

2020-10-26T11:27:15+01:00 {host="centos7", job="syslog"} Syslog connection established; fd='21', server='AF_INET(127.0.0.1:1514)', local='AF_INET(0.0.0.0:0)'

Accessing Loki from Grafana

Using containers, you can download and start Grafana with a single command:

docker run -d -p 3000:3000 grafana/grafana

This will start Grafana in a container. As you could guess it, the web interface is available on port 3000. The default user and password are admin/admin. Once you are logged in, you should configure Loki as a data source. Go to “Data sources” and select “Loki” from the list.

The web interface suggests using port 3100 on the localhost. As in our test setup, Loki runs directly on the host while Grafana is in a container, accessing the port on localhost from Grafana does not work. You should use the IP address of your host here. For example, in my case, the URL is:

http://172.16.167.137:3100/

Once you click “Save & Test”, you should see a green message at the bottom of your screen:

Data source connected and labels found.

Now go to “Explore”, and you are ready to query your logs. Using the same {job="syslog"} query in the query window, you should see the same logs as on the logcli command line.

What is next

As I already mentioned in the introduction, the setup detailed here is good enough for functionality testing. For performance testing you should make sure that logs are saved to a partition with enough disk space. You can also parse log messages in syslog-ng, store the results in structured data and use those as labels in Loki, but before doing that, make sure that you understand how labels work and what their limitations are.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.