Reading multiple files: wildcard file source in syslog-ng

Starting with version 3.10, syslog-ng can collect messages from multiple text files. You do not have to specify file names one by one, just use a wildcard to select which files to read. This is especially useful when you do not know the file names by the time syslog-ng is started. This is often the case with web servers with multiple virtual hosts. From this blog you can learn how to get started using the wildcard-file() source.

Before you begin

Before configuring syslog-ng, you should have a web server – or any other software writing multiple log files – already up and running. In my example I use Apache HTTPD access logs, but you should be able to adopt the basic example to any software easily just by changing file names.

Configuring syslog-ng

The following configuration reads any file ending with “log” in its name from the /var/log/apache2 directory and writes all messages in JSON format into a single file. You should append these configuration snippets to your syslog-ng.conf or in a separate .conf file under /etc/syslog-ng/conf.d/ if supported by your Linux distribution.

First, define a wildcard-file() source. There are two mandatory parameters:

• base-dir() configures the directory where syslog-ng looks for log files to read. In this case, it is the /var/log/apache2 directory.
• filename-pattern() accepts a simple glob pattern which defines files to search for. A “*” represents zero or more characters, while a “?” a single character. In this case, it is any file name that ends with “log”.

The no-parse flag is necessary in this example, because by default syslog-ng parses messages using the syslog parser, but Apache HTTPD uses its own format for logging. For a complete list of wildcard-file() options check the documentation at https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide.

source s_apache2 {
  wildcard-file(
    base-dir("/var/log/apache2/")
    filename-pattern("*log")
    flags(no-parse)
  );
};

Next, define a destination. Here, I’m using a JSON template, so that the different fields of the message are easy to distinguish. The “FILE_NAME” macro contains the file name with the full path name. The “MESSAGE” macro contains the whole message as it is read from the log files.

destination d_apache2 {
  file(
    "/var/log/web"
    template("$(format_json --key FILE_NAME --key MESSAGE)\n\n")
  );
};

Finally, define a log statement that connects the source and destination together:

log { source(s_apache2); destination(d_apache2); };

Save the configuration and reload it using “syslog-ng-ctl reload”.

Verifying your configuration

If you have configured file names correctly, you should see similar entries in /var/log/web (or the destination you have configured):

{"MESSAGE":"172.16.146.1 - - [14/Jun/2017:17:17:16 +0200] \"GET / HTTP/1.0\" 200 4 \"-\" \"w3m/0.5.3+git20170102\"","FILE_NAME":"/var/log/apache2/test_log"}

In the MESSAGE field you see a message in the Apache combined log format. If you use VirtualHost names in your file names, you can use this information to identify which log message belongs to which website.

What is next

Logging as a service (LaaS) providers often recommend their agents to be installed next to syslog(-ng) just to cover this situation. Installing additional software is not necessary any more to be able to forward messages from a directory of log files. Also, using LaaS providers from syslog-ng was never easier thanks to the syslog-ng configuration library (SCL), which hides away the complexity of setting up these destinations.

Try it yourself or check my blog next week where I will add a parser and a LaaS provider to the configuration.