Writing Python destination in syslog-ng: how to send log messages to MQTT

In my past two Python blogs I introduced you to the basics of the syslog-ng Python destination. In this blog I show you a working example of how you can publish your logs to MQTT using the Python destination of syslog-ng. If you are new to the Python destination you should start by reading the previous two parts that explain how to set up your environment and run your first Python application from syslog-ng:

• Getting started with the Python destination
• Learning the details: methods, options, starting on boot

Why an MQTT destination?

I was looking for a topic that I could use as a sample for demonstrating the Python destination in more depth. I wanted an easily available open source application with Python bindings. The next day someone was asking for an MQTT destination for syslog-ng, so my topic was born.

Disclaimer: “It works on my machine”. This was originally intended as a sample code. I do not use MQTT in production. There are limitations: no encryption or authentication. My blog contains the current version as of the time of writing. Pull requests are welcome, so you might find a slightly different code in my GitHub repository at https://github.com/czanik/syslog-ng-mqtt-dest.

Before you begin

First, you need an MQTT server. I used Mosquitto, which is open source and available in many Linux distributions. I installed it on a recent Fedora release and ran it with the default configuration.

On your syslog-ng machine you need a recent syslog-ng release with Python support enabled. I use 3.17.2, but with minimal modifications earlier versions might also work. My code was only tested with Python 3.

You also have to install the Paho MQTT Python client library. I installed the version available in Fedora. To install it on your machine, enter the following command:

dnf install python3-paho-mqtt

If it is not available in your distribution, you can also easily install it using pip:

pip install paho-mqtt

syslog-ng configuration

There are two parts of the syslog-ng Python destination. One is the actual Python code that syslog-ng runs and the other part is the syslog-ng configuration. In this section I show you the configuration part. You can append it to your syslog-ng.conf file, but I recommend saving the configuration in a separate .conf file under /etc/syslog-ng/conf.d/ if syslog-ng in your distribution is configured to use it.

destination d_python_to_mqtt {
    python(
        class("mqtt_dest.MqttDestination")
        options(
          # mandatory options
          host 127.0.0.1
          port 1883
          topic "syslog/warn"
          # optional options
          debug 1
          qos 2
        )
    );
};

The Python destination has a single mandatory option, the name of the class containing the methods that are called by syslog-ng. If your code is in an external file, you also have to include the name of the Python file without the .py extension. In the above example mqtt_dest.py was the original filename and MqttDestination is the name of the class, separated by a dot.

Using the options parameter is optional from the destinations point of view, but it makes your code more flexible and reusable. In the case of my MQTT destination three options are mandatory:

• host: the name or address of the MQTT server
• port: the port number of the MQTT server
• topic: the MQTT topic, where logs are published

There are also two optional options:

• debug: enter 0 for no debug messages, any positive integer for some extra debug messages. The default value is 0.
• qos: MQTT Quality of Service. The possible values are 0, 1 or 2, the default value is 0. Read the MQTT documentation for more details.

The Python code

In the following sections I show you the Python code in smaller chunks. At the end you will also see the complete code in one piece for your copy & paste convenience.

Getting started

"""
This is mqtt_dest.py, a sample syslog-ng Python destination saving logs to MQTT.
Use the "pip install paho-mqtt" command or install the relevant package from your distro.
Mandatory parameters: host, port and topic.
"""

import paho.mqtt.client as mqtt

class MqttDestination(object):
    """
    The MqttDestination class, reference this from syslog-ng.conf.
    """

I tend to start all my code with comments. Some consider it annoying, I consider it life-saving :) It contains some minimal information about what the file contains and also some requirements.

The line starting with import imports the Paho MQTT Python client library. The next line starting with class defines the MqttDestination class, which is referenced from the syslog-ng configuration.

Method: __init__()

    def __init__(self):
        """
        Initializes variables with default values.
        """
        self.host = None
        self.port = None
        self.topic = None
        self._is_opened = False
        self.debug = 0
        self.qos = 0
        self.mqttc = mqtt.Client("sng_mqtt")

The use of this method is not strictly necessary, you could include everything written here in the init() method. The reason I use it to reach a higher score when checking the code using pylint-3. I initialize everything here to its default value and create an MQTT client.

Method: printdebug()

    def printdebug(self, msg):
        """
        Prints debug message if debug is enabled.
        """
        if self.debug > 0:
            print(msg)

This is my own method to print debug messages if debug is enabled (debug > 0). Unlike all the others it is not called by syslog-ng directly.

Method: init()

    def init(self, options):
        """
        Initializes MQTT parameters.
        Fails if mandatory parameters are not available, or not in the right format
        """
        try:
            print('MQTT destination options: ' + str(options))
            self.host = options["host"]
            self.port = int(options["port"])
            self.topic = options["topic"]
            if "debug" in options:
                self.debug = int(options["debug"])
            if "qos" in options:
                self.qos = int(options["qos"])
        except Exception as err:
            print(err)
            print('MQTT destination: exiting in init()...')
            return False
        return True

This optional method is called by syslog-ng when syslog-ng is started or reloaded. When it returns with False, syslog-ng does not start. It can be used to check options and return False when they prevent the successful start of the destination.

The above code attempts to initialize the MQTT destination parameters. If a mandatory parameter is missing, or parameters are not in the right format, init() returns with False.

Method: is_opened()

    def is_opened(self):
        """
        Checks if destination is available
        """
        return self._is_opened

This optional method is called by syslog-ng after each send() call to check if the resource is still available. If it returns with False, syslog-ng calls the open() method to get access to the resource again. The _is_opened attribute is False by default. It is set to True in the open() method, if the resource – in our case the MQTT server – is accessible and ready to use. If open() or send() fails, it is set back to False.

Method: open()

    def open(self):
        """
        Opens connection to the MQTT server and start the loop.
        """
        try:
            self.mqttc.connect(self.host, self.port)
            self.mqttc.loop_start()
            self._is_opened = True
            self.printdebug('MQTT destination: opened...')
        except Exception as err:
            print(err)
            self.printdebug('MQTT destination: opening ' + self.host + ' at ' + str(self.port) + ' failed...')
            self._is_opened = False
            return False
        return True

This optional method is called by syslog-ng after init() to actually open the resource, that is in our case the MQTT server. It is also called by syslog-ng if open() or send() returns False.

In the try part of the method, it attempts to connect to the MQTT server on the given host and port. Next, the MQTT client event handler is started and the _is_opened attribute is set to True.

If the connection fails an error message is printed and open() returns with False.

Method: close()

    def close(self):
        """
        Closes the connection.
        """
        self.mqttc.disconnect()
        self.printdebug('MQTT destination: closing connection to ' + self.host + ' at ' + str(self.port))
        self._is_opened = False

The close() method is the pair of open(). This optional method is called by syslog-ng right before deinit() when stopping or reloading syslog-ng. It is also called when send() fails and syslog-ng calls a close() and open() pair before trying to send the message again.

Once the connection to the MQTT server is closed the _is_opened attribute is also set to False.

Method: deinit()

    def deinit(self):
        """
        pair of init(), normally empty
        """
        pass

This optional method is called by syslog-ng after close() when stopping or reloading syslog-ng. It is the pair of init() and is normally empty.

Method: send()

    def send(self, msg):
        """
        Sends the message.
        """
        decoded_msg = msg['MESSAGE'].decode('utf-8')
        try:
            self.printdebug('MQTT destination: before sending')
            self.mqttc.publish(self.topic, decoded_msg, qos=self.qos)
            self.printdebug('MQTT destination: after sending')
        except Exception as err:
            print(err)
            self.printdebug('MQTT destination: sending to topic ' + self.topic + ' failed...')
            self._is_opened = False
            return False
        return True

In theory, send() is the only method required by syslog-ng. In practice, I was able to create a simple destination in Python where initialization, sending data, closing connection were all implemented in a single send() method. In real-life circumstances, it is not really practical to do so and your code is a lot more robust if you use the optional methods too.

As I have already written earlier: the send() method attempts to send the message to the destination, that is in this case to an MQTT server. If it fails the _is_opened attribute is set to False and the method returns False.

Testing

If you use the “syslog/warn” topic in your configuration – as seen in my sample syslog-ng configuration – you can use the mosquitto_sub command on the server to check if you have any incoming messages:

mosquitto_sub -t 'syslog/warn' -v

Without syslog-ng

You can test part of the above Python code even without syslog-ng. Just append the following snippet to your code:

# Code to test the MqttDestination outside of syslog-ng.
print('starting up...')
bla = MqttDestination()
bla.init(
    options=dict(
        host="172.16.167.132",
        port="1883",
        topic="syslog/warn",debug="1"
    )
)
bla.open()
bla.send(msg=dict(MESSAGE=b"It's working..."))
bla.send(msg=dict(MESSAGE=b"Still working..."))
bla.close()
bla.deinit()

Edit the options, especially the host parameter, to fit your environment. Once you run mqtt_dest.py, you should see the two messages that are defined in msg=dict() appearing on the terminal where you run mosquitto_sub.

With syslog-ng

If you do not know how to run Python code from syslog-ng, check my previous two blogs that I have linked to from the introduction. Use the syslog-ng configuration from the beginning of this post, but make sure that the host parameter points to the host in your environment. Also make sure that there are actually logs directed at it. For example:

log {
    source(s_sys);
    destination(d_python_to_file);
};

When you (re)start syslog-ng with the new configuration, you should see your log messages showing up in the output of mosquitto_sub.

The whole Python code for your copy & paste convenience

"""
This is mqtt_dest.py, a sample syslog-ng Python destination saving
logs to MQTT
Use "pip install paho-mqtt" or install the relevant package from your distro
host, port and topic are mandatory parameters
"""

import paho.mqtt.client as mqtt


class MqttDestination(object):
    """
    the MqttDestination class, reference this from syslog-ng.conf
    """

    def __init__(self):
        """initializing variables with default values"""
        self.host = None
        self.port = None
        self.topic = None
        self._is_opened = False
        self.debug = 0
        self.qos = 0
        self.mqttc = mqtt.Client("sng_mqtt")

    def printdebug(self, msg):
        """prints debug message if debug is enabled"""
        if self.debug > 0:
            print(msg)

    def init(self, options):
        """
        initializes MQTT parameters, fails if mandatory parameters
        are not available, or not in the right format
        """
        try:
            print('MQTT destination options: ' + str(options))
            self.host = options["host"]
            self.port = int(options["port"])
            self.topic = options["topic"]
            if "debug" in options:
                self.debug = int(options["debug"])
            if "qos" in options:
                self.qos = int(options["qos"])
        except Exception as err:
            print(err)
            print('MQTT destination: exiting in init()...')
            return False
        return True

    def is_opened(self):
        """Checks if destination is available"""
        return self._is_opened

    def open(self):
        """
        opens connection to the MQTT server and start the loop
        """
        try:
            self.mqttc.connect(self.host, self.port)
            self.mqttc.loop_start()
            self._is_opened = True
            self.printdebug('MQTT destination: opened...')
        except Exception as err:
            print(err)
            self.printdebug('MQTT destination: opening ' + self.host + ' at ' + str(self.port) + ' failed...')
            self._is_opened = False
            return False
        return True

    def close(self):
        """
        closes the connection
        """
        self.mqttc.disconnect()
        self.printdebug('MQTT destination: closing connection to ' + self.host + ' at ' + str(self.port))
        self._is_opened = False

    def deinit(self):
        """
        pair of init(), normally empty
        """
        pass

    def send(self, msg):
        """
        sends the message
        """
        decoded_msg = msg['MESSAGE'].decode('utf-8')
        try:
            self.printdebug('MQTT destination: before sending')
            self.mqttc.publish(self.topic, decoded_msg, qos=self.qos)
            self.printdebug('MQTT destination: after sending')
        except Exception as err:
            print(err)
            self.printdebug('MQTT destination: sending to topic ' + self.topic + ' failed...')
            self._is_opened = False
            return False
        return True

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/balabit/syslog-ng. On Twitter, I am available as @PCzanik.