Syslog-ng 3.33: the MQTT destination

Version 3.33 of syslog-ng introduced an MQTT destination. It uses the paho-c client library to send log messages to an MQTT broker. The current implementation supports version 3.1 and 3.1.1 of the protocol over non-encrypted connections, but this is only a first step.

From this blog, you can learn how to configure and test the mqtt() destination in syslog-ng.

Before you begin

To use the MQTT destination of syslog-ng, you need to use at least syslog-ng version 3.33. It is not yet included in most Linux distributions, but luckily it is available in some 3^rd-party syslog-ng repositories. You can find a list of them at https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx MQTT support is usually included in a sub-package of syslog-ng, often called syslog-ng-mqtt. You also need an MQTT broker, where you can forward log messages using the mqtt() destination.,

For my tests, I used Fedora 34 with the Mosquitto MQTT broker and syslog-ng installed from the unofficial Copr repository.

Configuring syslog-ng

If your distribution of choice supports it, create a new configuration file under the /etc/syslog-ng/conf.d/ directory with a .conf extension. Otherwise, append the configuration below to syslog-ng.conf.

destination d_mqtt {
  mqtt (
    address("tcp://localhost:1883"),
    topic("test/$HOST"),
    fallback-topic("syslog/fallback")
    template("$MESSAGE")
    qos(1)
  );
};

log {
    source(s_sys);
    destination(d_mqtt);
};

The MQTT destination has three mandatory options:

• address() defines where to send log messages: it includes the hostname (or IP address) and the port number of the MQTT broker. These values work if the broker is running on localhost at the default port.

• topic() defines in which topic syslog-ng stores the log message. You can also use templates here, and use, for example, the $HOST macro in the topic name hierarchy.

• fallback-topic() is used when syslog-ng cannot post a message to the originally defined topic (which can include invalid characters coming from templates).

Optional parameters of the MQTT destination include:

• template(), where you can configure the message template sent to the MQTT broker. By default, the template is: “$ISODATE $HOST $MSGHDR$MSG”

• qos stands for quality of service and can take three values in the MQTT world. Its default value is 0, where there is no guarantee that the message is ever delivered. Setting it to 1 makes sure that the message is delivered at least once, while 2 ensures that a message is delivered exactly once. Obviously, 0 has the best performance, while 2 can be much slower.

The log statement connects the local log sources with the MQTT destination. Note, that the name of the source might be different, depending on syslog-ng.conf. Configurations on Fedora and RHEL call the local log source s_sys, by default.

Testing

As mentioned in the introduction, I used the Mosquitto MQTT broker for testing. There are dozens of others listed at https://mqtt.org/software/, verifying incoming messages is different for each software.

Once you reloaded syslog-ng to ensure that the configuration takes effect, you are ready for testing. We use two utilities for testing:

• logger sends syslog messages to the local server

• mosquitto_sub is part of the Mosquitto MQTT broker software and can subscribe to messages arriving on a broker

First start mosquitto_sub in a terminal:

mosquitto_sub -h localhost -p 1883 -t 'test/+'

This will listen to incoming messages on the broker on all subtopics under the “test” topic. “+” is a wildcard here.

Next, you can send some log messages using the logger command:

logger this is a test

And you should see the line appear in the output of mosquitto_sub:

2021-07-26T16:39:55+02:00 fedora34.localdomain root[3077]: this is a test

What is next?

This is just the first step in adding MQTT support to syslog-ng. Your feedback is very welcome about this new feature. Detailed problem reports help to make the next version more robust while positive feedback lets us know that the feature is in use and it is worth developing it further.

-

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @Pczanik.