Enabling PCRE dupnames in syslog-ng

One of the major syslog-ng features is that it can parse log messages and create name-value pairs from them. Until now the PCRE parser could not handle duplicate names for named subpatterns. Version 3.29 of syslog-ng resolves this issue by adding the “dupnames” flag.

From this blog you can learn why the dupnames flag is important and how you can enable and test it.

Why is it important?

In this blog I used an example regular expression from the PCRE documentation: https://www.pcre.org/original/doc/html/pcrepattern.html#SEC16 It finds English day names in lines of text and saves the first three characters into a named subpattern. There is not much use to store it under different names, so the expression uses the same name, “DN” for all subpatterns. When you insert that expression in a syslog-ng configuration and test it, it will fail loudly:

localhost:~ # syslog-ng -s
Error parsing filter expression, compiling the regexp failed, error=Failed to compile PCRE expression >>>(?<DN>Mon|Fri|Sun)(?:day)?|(?<DN>Tue)(?:sday)?|(?<DN>Wed)(?:nesday)?|(?<DN>Thu)(?:rsday)?|(?<DN>Sat)(?:urday)?<<< `two named subpatterns have the same name' at character 32 in /etc/syslog-ng/conf.d/pcre_dupnames.conf:2:9-2:121:
1       filter f_dupnames {
2----->   match("(?<DN>Mon|Fri|Sun)(?:day)?|(?<DN>Tue)(?:sday)?|(?<DN>Wed)(?:nesday)?|(?<DN>Thu)(?:rsday)?|(?<DN>Sat)(?:urday)?" value(MSG) flags(store-matches));
2----->         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3       };
4       
5       destination f_file {
6         file("/var/log/file" template("And abbr of the day is: $DN\n"));
7       };

Included from /etc/syslog-ng/syslog-ng.conf:224:1-224:1:

Starting with syslog-ng version 3.29 you can use the “dupnames” flag for PCRE expressions to allow duplicate names for subpatterns. You should enable this only when you are sure, that the given expression matches only once.

Before you begin

As it was already mentioned, this feature was introduced in syslog-ng version 3.29. If this version is not yet available in your Linux distribution of choice, check https://www.syslog-ng.com/3rd-party-binaries if it is available in a 3^rd party repository.

Configuration

This configuration finds day names and stores the first three letters in a subpattern, or – in syslog-ng terminology – in a name-value pair. It writes the three letter abbreviation into a file called /var/log/file with a simple template referring to the name-value pair extracted using the regular expression.

filter f_dupnames {
  match("(?<DN>Mon|Fri|Sun)(?:day)?|(?<DN>Tue)(?:sday)?|(?<DN>Wed)(?:nesday)?|(?<DN>Thu)(?:rsday)?|(?<DN>Sat)(?:urday)?" value(MSG) flags(store-matches,dupnames));
};

destination f_file {
  file("/var/log/file" template("And abbr of the day is: $DN\n"));
};

log {
    source(src);
    filter(f_dupnames);
    destination(f_file);
};

In most Linux distros you can drop this configuration with a .conf extension under /etc/syslog-ng/conf.d/. In other cases append this configuration snippet to syslog-ng.conf. Make sure that the source in the log path is the same that you have in syslog-ng.conf for local logs. In this configuration the source is called “src” (just as in openSUSE & SLES), in Fedora & RHEL the default syslog-ng.conf calls this source as “s_sys”.

Testing

Once you saved your configuration and reloaded syslog-ng, you are ready for testing. It is easy, just send three messages using logger and observe the results:

localhost:~ # logger Hello World
localhost:~ # logger The first day of the week is Monday
localhost:~ # logger Thanks God, it\'s Friday

In most syslog-ng configurations you can find the majority of log messages in a file called: /var/log/messages. Check the end of this file and then /var/log/file for the lines with the abbreviations of the extracted day names.

localhost:~ # tail -3 /var/log/messages
Sep 21 09:29:33 localhost root[3285]: Hello World
Sep 21 09:29:42 localhost root[3287]: The first day of the week is Monday
Sep 21 09:29:48 localhost root[3288]: Thanks God, it's Friday
localhost:~ # tail -3 /var/log/file
And abbr of the day is: Mon
And abbr of the day is: Fri

Of course on a busy syslog server the first file might contain other lines at the end instead of those we just sent in. In this case you could check the file more in depth. But it is not really necessary, as the focus is on the other file. From this you can see that the filter worked: only the lines with day names are saved using the formatting from the template.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.