Last week you learned the very basics of the syslog-ng Python destination. This time, you will move a bit further and learn about a few more configuration options and optional methods. If you are new to the Python destination and want to get started with it, read the first part of the series to get started. It is available at https://www.syslog-ng.com/community/b/blog/posts/python-destination-getting-started.
This blog is loosely based on the Python chapter of the syslog-ng GitBook, a documentation for developers: https://syslog-ng.gitbooks.io/getting-started/content/chapters/chapter_5/section_1.html
Before you begin
The Python destination of syslog-ng evolved quite a lot since it first appeared. There were a number of smaller and larger changes, for example, its configuration and the mandatory parameters. Originally only version 2.7 of the language was supported, now Python 3 works as well.
I used syslog-ng version 3.17 and Python version 3.4 to write this blog, but my examples are also tested to work with Python 2.7.
By the time I am writing this blog only Fedora Rawhide features syslog-ng 3.17. For other platforms check https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx for up-to-date 3rd party syslog-ng packages. Note that support for Python is often not included in the base syslog-ng package, but in a sub-package. In openSUSE and Fedora it is called syslog-ng-python. On FreeBSD you have to recompile the port yourself with Python support enabled.
Configuration options
Last week you already learned about using the one and only mandatory class() option and also passing name-value pairs to the Python code using value-pairs() or without it. The following configuration is extended with a few more possibilities:
destination d_python_to_file {
python(
class("pythonexample_full.TextDestination")
on-error("fallback-to-string")
value-pairs(scope(rfc5424))
options(
opt1 "first option"
opt2 second
)
);
};
log {
source(src);
destination(d_python_to_file);
};
You can use the on-error() option together with value-pairs(). It specifies what happens when a message cannot be parsed properly. You can read more about the possible parameters among the MongoDB destination options in the syslog-ng documentation at https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.17/administration-guide/38. By default the log message is dropped and an error is logged about it to the internal() source.
Finally, you can pass options from the syslog-ng configuration to the Python application. This can be used for example to configure the server name and port number.
Sample Python app
Below you can read a Python app that is a bit more complex and a lot more elegant than the one in the previous Python post. You can use it as a template for your own projects. It contains not only the send() method, but all optional methods too. If you follow the naming conventions from the configuration example above, save this file under the pythonexample_full.py file name.
class LogDestination(object):
def open(self):
"""Open a connection to the target service
Should return False if opening fails"""
return True
def close(self):
"""Close the connection to the target service"""
pass
def is_opened(self):
"""Check if the connection to the target is able to receive messages"""
return True
def init(self, options):
"""This method is called at initialization time
Should return false if initialization fails"""
return True
def deinit(self):
"""This method is called at deinitialization time"""
pass
def send(self, msg):
"""Send a message to the target service
It should return True to indicate success. False will suspend the
destination for a period specified by the time-reopen() option."""
return True
class TextDestination(LogDestination):
def __init__(self):
self.outfile = None
self._is_opened = False
def init(self, options):
self.outfile = open("/tmp/example.txt", "a")
self.outfile.write("initialized with {}\n".format(options))
self.outfile.flush()
return True
def is_opened(self):
return self._is_opened
def open(self):
self.outfile.write("opened\n")
self.outfile.flush()
self._is_opened = True
return True
def close(self):
self.outfile.write("closed\n")
self.outfile.flush()
self._is_opened = False
def deinit(self):
self.outfile.write("deinit\n")
self.outfile.flush()
self.outfile.close();
def send(self, msg):
self.outfile.write("Name Value Pairs are:\n")
for key,v in msg.items():
self.outfile.write(str(key) + " = " + str(v) + "\n");
self.outfile.write("________________________\n\n")
self.outfile.flush()
return True
Here I will show you a sample output. Once syslog-ng was started, I sent it a single log message (“bla”) using the command “logger bla” and stopped syslog-ng using ^C. The results from /tmp/example.txt are the following:
initialized with {'opt1': 'first option', 'opt2': 'second'}
opened
Name Value Pairs are:
PROGRAM = b'root'
HOST = b'linux-6chp'
PRIORITY = b'notice'
PID = b'9335'
FACILITY = b'user'
DATE = b'Sep 4 13:28:40'
MESSAGE = b'bla'
________________________
closed
deinit
In the first line you can see that the destination is initialized and options configured in syslog-ng are passed on to the Python code. In the second line you can see that the file is opened. Subsequent lines list the name-value pairs as specified by the value-pairs() option. And finally – once ^C is pressed – the file is closed and the destination is deinited.
Optional methods
Comments in the sample Python app code explain most, but not all of the working details. In this section you can learn more about the function of each method.
- init() is run when syslog-ng is started or reloaded. When it returns with False, syslog-ng does not start. It can be used to check options and return False when they prevent the successful start of the destination.
- open() is used to open the resource belonging to the destination. It is called after init() when syslog-ng is started or reloaded. If send() returns with an error, syslog-ng calls a close() and open() pair before trying to send again.
If a message cannot be delivered after a predefined number of retries() (by default: 3), the message is discarded and syslog-ng continues with the next message.
If open() fails, it is retried in time-reopen() (by default: 1 second for the Python destination) intervals. The value of time-reopen() is not inherited from the global option. - deinit() is the pair of init(). It is called before stopping and reloading syslog-ng and does not return a value.
- close() is the pair of open(). Usually it is called right before deinit() when stopping or reloading syslog-ng. It is also called when send() fails and syslog-ng calls a close() and open() pair before trying to send the message again.
Starting syslog-ng on boot
Until now we have always started syslog-ng from the command line, which is fine for testing our freshly written Python applications, but not for production use. If you recall, the Python destination requires the PYTHONPATH environment variable to be configured. Using “export” before starting syslog-ng works from your terminal window, but not when starting up a system. The solution for configuring PYTHONPATH varies from system to system.
openSUSE / SLES
The systemctl command sources the /etc/sysconfig/syslog file before starting syslog-ng. You can append the following line to the end of the file to ensure that syslog-ng starts your Python app:
PYTHONPATH="/etc/syslog-ng/py"
If you do not want to follow the naming conventions in this post, change the path to the one you actually use.
Fedora / RHEL & CentOS
The systemctl command sources the /etc/sysconfig/syslog-ng file before starting syslog-ng. You can append the following line to the end of the file to ensure that syslog-ng starts your Python app:
PYTHONPATH="/etc/syslog-ng/py"
If you do not want to follow the naming conventions in this post, change the path to the one you actually use.
FreeBSD
If you use FreeBSD, the first step is to compile syslog-ng from ports with Python support enabled. Once that is ready you can append PYTHONPATH to /etc/rc.conf. Because there is no dedicated syslog-ng subdirectory under /etc on FreeBSD, I put the Python code under my home directory:
PYTHONPATH="/home/czanik/py"
If you do not want to follow the naming conventions in this post, change the path to the one you actually use.
By now you should be able to code and use the Python destination in syslog-ng.
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/balabit/syslog-ng. On Twitter, I am available as @PCzanik.
