Recently, a number of quite complex configurations came up while syslog-ng users were asking for advice. Some of these configurations were even pushing the limits of syslog-ng (regarding the maximum number of configuration objects). As it turned out, these configurations could be significantly simplified using the in-list() filter, one of syslog-ng’s lesser known features.
First, a bit of history. The idea of the in-list() filter came to me while I was listening to Xavier Mertens at a Libre Software Meeting conference talk in France. In his talk, he described how to check log messages for suspicious IP addresses. He used free IP address lists from the Internet (spammer IP addresses, malware command and control IP addresses, etc.) and, using a batch process, he kept checking if any of those were present in the log messages on a nightly basis.
It occurred to me that all of the above could be done in real-time. Namely, several different parsers capable of extracting IP addresses and other important information from log messages as they arrive are already available in syslog-ng. All that was missing was a tool that could compare the extracted value with a list of values coming from a file. This tool was implemented quickly as a ‘spare time project’ by one of my colleagues. This is how the in-list() filter was born.
As you can see, the original use case came from IT security. Still, the first real world use I know about came from an operation team where they wanted to forward logs based on application name lists. This later became the example configuration in the documentation: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.22/administration-guide/inlist Previously, you had to maintain a long list of filters in your syslog-ng configuration and connect them using a Boolean OR. Using the in-list() filter, the list is moved into an easy-to-maintain external file, where each value is listed on a new line. Note that the values in this file are case sensitive, so Bla, bLa, and blA are all different.
We had reports that some syslog-ng users were generating syslog-ng.conf using a script. In these cases, the configuration became overly long and eventually reached the number of maximum possible objects. In comparison with using a (possibly very long) list of filters, using the in-list() filter has many advantages:
-
logic and data are separated
-
less error-prone, as there is no need to edit the configuration file each time
-
easier to maintain, as you edit a list of values instead of a configuration file
-
better performance
-
the number of values in the file used by the in-list() filter is only limited by your available RAM
There is no exact limit to specify when you should change from a list of filters to the in-list() filters. I would suggest using in-list() as soon as your configuration becomes difficult to read.
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/balabit/syslog-ng. On Twitter, I am available as @PCzanik.
