Version 3.35.1 of syslog-ng introduced an MQTT source. Just for some fun in the last syslog-ng blog post of the year, I created an endless loop using syslog-ng and the Mosquitto MQTT broker. Of course, it does not have much practical value other than possibly a bit of stress testing, but hopefully provides a fun introduction to MQTT-related technologies in syslog-ng.
Before you begin
As already mentioned in the introduction, MQTT support on the source side arrived in version 3.35.1 of syslog-ng. The MQTT destination arrived in version 3.33 and 3.34, but that alone is not enough to create an endless loop :-) You also need an MQTT broker. I used Mosquitto without any security measures for simplicity, but any other implementation should work as well. The example configuration and the tool I show to send the initial message to MQTT are from Mosquitto.
Configuring Mosquitto and syslog-ng
To keep things simple, I did not use any security for MQTT: no username and password, no encryption. My Mosquitto configuration was very simple:
listener 1883 0.0.0.0 allow_anonymous true
This allowed a very simple configuration on the syslog-ng side as well. I created the following configuration under the /etc/syslog-ng/conf.d/ directory as mqtt.conf (if syslog-ng on your host does not support including configuration files from this directory, simply append it to syslog-ng.conf)
source s_mqtt {
mqtt(
address("tcp://localhost:1883")
topic("mytopic")
);
};
destination d_mqtt {
mqtt(
address("tcp://localhost:1883")
topic("mytopic")
template("$MESSAGE")
);
};
#destination d_frommqtt {
# file("/var/log/frommqtt");
#};
log {
# source(s_sys);
source(s_mqtt);
destination(d_mqtt);
# destination(d_frommqtt);
};
Well, OK. The syslog-ng configuration is slightly more than the bare minimum. However, the extra parts are commented out. You can use those for debugging if you need/want them by removing the hash signs from the beginning of the lines.
The configuration consists of four blocks, but only three is in use. The fourth one is commented out. It is a file destination and can be used to double check that messages arrive as expected.
The first entry is an MQTT source. It connects to the MQTT broker on the localhost and collects log messages from a topic called “mytopic”.
The second entry is an MQTT destination. It also connects to the MQTT broker on the localhost and sends log messages to the very same topic as the source. The template is there to make sure that the message length does not grow each time syslog-ng sends it back to MQTT.
If you want to check that messages really arrive from the MQTT source, you should take out the file destination from comments and also uncomment it in the logpath.
Finally, the log path connects the MQTT source and destinations together. Only these two are enabled, but you can do additional tests by removing the hash mark from the beginning of the lines and commenting out others.
Testing
Once you saved the configuration and loaded it to syslog-ng, you are ready for testing. Anything arriving on the MQTT source is immediately sent back to the same topic, creating an endless loop. All you need is a single message sent to the topic. If you use the Mosquitto broker, as I did, then it comes bundled with a few utilities. One of them can post messages from the command line:
mosquitto_pub -h localhost -p 1883 -t mytopic -m test
If you want longer test messages using multiple words, you need to use quotes around the message:
mosquitto_pub -h localhost -p 1883 -t mytopic -m “this is a test”
A few minutes after the message is sent, you should see the CPU consumption go up both for syslog-ng and Mosquitto.
What is next?
Of course, endless loops are good for stress testing and some fun, but do not have much practical value. However, I hope, that this example configuration inspired you to try the MQTT source of syslog-ng. You can learn more about the MQTT source from this PR: https://github.com/syslog-ng/syslog-ng/pull/3809 And read my earlier blogs about the MQTT destination: https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-3-34-mqtt-destination-with-tls-and-websocket-support
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.