You can store your log messages to many different destinations using syslog-ng, but of course not everywhere. This is where the Python destination of syslog-ng can come handy. You can extend syslog-ng easily with your own code written in Python and store your log messages to the destination of your choice.
From this blog you can learn the very basics of writing your first syslog-ng destination in Python. It is enough to setup and test your environment. From my next post you can learn about further methods and configuration possibilities to make your Python code production ready.
Before you begin
The Python destination of syslog-ng evolved quite a lot since it first appeared. There were a number of smaller and larger changes, for example, its configuration and the mandatory parameters. Originally only version 2.7 of the language was supported, now Python 3 works as well.
I used syslog-ng version 3.17 and Python version 3.4 to write this blog, but my examples are also tested to work with Python 2.7.
By the time I am writing this blog only Fedora Rawhide features syslog-ng 3.17. For other platforms check https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx for up-to-date 3rd party syslog-ng packages. Note that support for Python is often not included in the base syslog-ng package, but in a sub-package. In openSUSE and Fedora it is called syslog-ng-python. On Debian and Ubuntu the package is called syslog-ng-mod-python. On FreeBSD you have to recompile the port yourself with Python support enabled.
How it works?
The Python interpreter runs inside syslog-ng, which has several advantages. You could already run Python applications from syslog-ng using the program() destination for several years, but using the Python destination simplifies many things. You have easy access to Python error messages through the internal() source of syslog-ng. Accessing name-value pairs of syslog-ng is also easier. You do not have to package them nicely on the syslog-ng side into JSON or any other format and then extract them on the Python side again. Instead, name-value pairs can now be accessed directly through a message object or a dict.
You can embed a simple Python code even within syslog-ng.conf. For more complex applications you can save Python applications into separate files.
The destination has two parts. The first part is a regular destination declared in the syslog-ng configuration. The other part is the code. As I wrote above, this can be included – in a separate section – inside the configuration file or can also be stored in regular Python files.
Inline code
I will start with the simplest possible configuration and code. Here I do not utilize any optional features of the destination or value-pairs(), just store any incoming message into a file using a hard coded file name. The code is also included within the syslog-ng configuration.
destination d_python_to_file {
python(
class("TextDestination")
);
};
log {
source(src);
destination(d_python_to_file);
};
python {
class TextDestination(object):
def send(self, msg):
self.outfile = open("/tmp/example.txt", "a")
self.outfile.write("MESSAGE = %s\n" % msg["MESSAGE"])
self.outfile.flush()
self.outfile.close();
return True
};
As you can see, the Python destination has a single mandatory parameter: the class name implementing the destination.
Here, the Python code is part of the syslog-ng configuration. The class implementing the destination is enclosed in a python {}; statement. Within that you must implement a single method (there are many more, but only this one is mandatory):
send(self, msg)
When you are not using value-pairs() in the configuration, your code receives a message object where you can refer to the different name-value pairs using the index operator. For example, msg[“MESSAGE”] returns with the value of $MESSAGE.
Code in external file
The other possibility is to store the Python code in a separate file. It requires some extra preparation, because to find your code, the Python interpreter requires the PYTHONPATH environment variable configured. When you start syslog-ng from your (bash) shell and your code is saved under the /etc/syslog-ng/py directory you can set it up using the following command:
export PYTHONPATH=$PYTHONPATH:/etc/syslog-ng/py
The related part of the syslog-ng configuration is the following:
destination d_python_to_file {
python(
class("pythonexample.TextDestination")
value-pairs(key(MESSAGE))
);
};
log {
source(src);
destination(d_python_to_file);
};
There are a couple of changes compared to the previous configuration:
- The class name in this example also includes the name of the Python file containing the class – without the .py extension.
- The Python code is not part of the syslog-ng configuration.
- The configuration snippet uses value-pairs() to select which name-value pairs are passed on to the Python code. (Note: this change is not related to storing code in a separate file.)
You can store the Python code in any directory in your system. I use /etc/syslog-ng/py to keep the code together with the configuration. The content of the pythonexample.py file from that directory is the following:
class TextDestination(object):
def send(self, msg):
self.outfile = open("/tmp/example.txt", "a")
for key,v in msg.items():
self.outfile.write(str(key) + " = " + str(v) + "\n");
self.outfile.write("________________________\n\n")
self.outfile.flush()
self.outfile.close();
return True
When you start syslog-ng from the command line for testing (syslog-ng -Fvde) you will see similar output as with our first sample program, except that in this case you can see a horizontal line after each log message. It is not strictly necessary if you have a single name-value pair to print. But if you experiment a bit with the value-pairs option in the syslog-ng configuration to include more fields you will appreciate it. Especially with value-pairs(scope(everything)) :-)
I hope this blog made you to try out the Python destination of syslog-ng. In my next blog I will introduce you:
- more configuration options for the Python destination
- optional methods for the Python class
- how to start syslog-ng using systemctl on RPM distros when using a Python destination
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/balabit/syslog-ng. On Twitter, I am available as @PCzanik.