Upgrading a syslog-ng PE 6 configuration to 7

As we learned in my previous blog, upgrading syslog-ng PE 6 to 7 is no more difficult than any major version upgrade. The key here is to use the latest version of PE 7 (at least version 7.0.17), as it has the necessary compatibility fixes to make the upgrade experience smoother. From this blog, you can learn how to upgrade a simple configuration containing a wildcard file source and an RLTP source to work correctly with syslog-ng PE 7.

Before you begin

First of all, make sure that you read my previous blog, which gives you a general overview of the upgrade process: https://www.syslog-ng.com/community/b/blog/posts/upgrading-syslog-ng-pe-from-version-6-to-7. Also, make sure that you download version 7.0.17 or later. I can not emphasize enough how important the version number is, and I admit that

even I downloaded the wrong version while I was writing my first upgrade blog :-)

A simple sample configuration

You can find a very simple sample configuration below. Beyond the default PE configuration, it also contains two of the most typical features in PE that need some extra work while upgrading. You can find the syslog-ng PE configuration at /opt/syslog-ng/etc/syslog-ng.conf:

[root@centos7 ~]# cat /opt/syslog-ng/etc/syslog-ng.conf
@version: 6.0
#Default configuration file for syslog-ng.
#
# For a description of syslog-ng configuration file directives, please read
# the syslog-ng Administrator's guide at:
#
# https://www.balabit.com/support/documentation
#
@include "scl.conf"

options {
};

######
# sources
source s_local {
# message generated by Syslog-NG
internal();
system();
};


######
# destinations
destination d_messages { file("/var/log/messages"); };


log {
source(s_local);

destination(d_messages);
};

# wildcard file source (reading http server logs)
source s_file { file("/var/log/httpd/*" follow-freq(1));};
destination d_file { file("/var/log/allweb.txt");};
log {source(s_file); destination(d_file);};

# RLTP server
source s_rltp { syslog(port(601) transport(rltp(tls-required(no))));};
destination d_fromrltp { file("/var/log/fromrltp.txt");};
log {source(s_rltp); destination(d_fromrltp);};

Obviously, a real configuration is a lot more complex as it also has filters, parsers, uses encryption over the network, and so on. Still, adding those elements would not really add any value here, and they only made the configuration and description more difficult to follow.

Updating the configuration

Unless extended down time is not a problem in your environment, you should do the configuration upgrade and testing in a separate environment. This way, upgrading your production server takes just a few seconds, as you have a ready to use configuration.

So, copy the above syslog-ng PE 6.0 configuration to a test environment where you installed syslog-ng PE 7.0.17+. Make sure that your syslog-ng PE 7 license file (license.txt) is also copied to /opt/syslog-ng/etc if you want to test server features, like the included RLTP (now renamed to ALTP), for example.

If you do a syntax check on your configuration, you will see a few warnings and an error as well:

[root@centos7 ~]# /opt/syslog-ng/sbin/syslog-ng -s
[2019-11-26T16:20:47.298400] WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode. Please update it to use the syslog-ng PE 7.0 format at your time of convenience. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file.;
[2019-11-26T16:20:47.363130] WARNING: Using wildcard characters in the file() source is deprecated, use wildcard-file() instead. The legacy wildcard file() source can only monitor up to 100 files, use wildcard-file(max-files()) to change this limit;
Error parsing afsocket, syntax error, unexpected '(', expecting ')' in /opt/syslog-ng/etc/syslog-ng.conf:40:48-40:49:
35      source s_file { file("/var/log/httpd/*" follow-freq(1));};
36      destination d_file { file("/var/log/allweb.txt");};
37      log {source(s_file); destination(d_file);};
38      
39      # RLTP server
40----> source s_rltp { syslog(port(601) transport(rltp(tls-required(no))));};
40---->                                                ^
41      destination d_fromrltp { file("/var/log/fromrltp.txt");};
42      log {source(s_rltp); destination(d_fromrltp);};
43      

Let’s start with the easy ones:

• update the version number from 6.0 to 7.0

• rename rltp() to altp()

The first line should look like this:

@version: 7.0

And line 40 ( the one with the rltp() source ) should look like this:

source s_rltp { syslog(port(601) transport(altp(tls-required(no))));};

Note that I changed only a single character, where it mattered from the configuration’s point of view. I did not rename the source, only the driver name.

When you do a syntax check now, the first warning and the error disappear:

[root@centos7 ~]# /opt/syslog-ng/sbin/syslog-ng -s
[2019-11-26T16:37:46.715187] WARNING: Using wildcard characters in the file() source is deprecated, use wildcard-file() instead. The legacy wildcard file() source can only monitor up to 100 files, use wildcard-file(max-files()) to change this limit;

As you can see, it is a warning. What it means is that with some limitations, your configuration works. With this configuration, syslog-ng can monitor up to a hundred files.

If you like to live dangerously and have less than a hundred files to monitor by the wildcard file source, you can stop here and use this configuration in production.

However, note that there are multiple drawbacks:

• the old syntax is deprecated and will be removed in a future version

• if the number of files to monitor grows silently above a hundred, you will start losing log messages

Finishing up: wildcard file source

Originally, the syslog-ng PE file source also handled wildcards in file names. While a limited compatibility remained to make upgrades easier, there is now a new, dedicated driver to handle wildcards in file names. It is called the wildcard-file() source. You can read more in depth about it in the documentation at: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/release-notes/wildcard-file-source-options

Here is the new s_file source, using the new wildcard-file() driver to collect http server logs:

source s_file { wildcard-file(
        base-dir("/var/log/httpd/")
        filename-pattern("*")
        recursive(no)
        follow-freq(1)
        max-files(200)
    );
};

Of course, you can still write it on a single line, I only put it on multiple lines for readability. If you do a syntax check now, you will see that the configuration is now really ready for the prime time:

[root@centos7 ~]# /opt/syslog-ng/sbin/syslog-ng -s
[root@centos7 ~]#

Going to production

Before going into production, you should also test your configuration. In the above case, browse the local webserver and check if logs appear in the /var/log/allweb.txt file. You can test ALTP using the loggen utility of syslog-ng:

/opt/syslog-ng/bin/loggen -L localhost 601

Check the results in the /var/log/fromrltp.txt file.

Once everything works as expected, copy the freshly created syslog-ng.conf to your syslog-ng PE 6.0 server. Also copy the new license.txt to the /opt/syslog-ng/etc directory. You can now update PE 6 to PE 7. On an RHEL system it is as simple as:

rpm -Uvh syslog-ng-premium-edition-7.0.17-1.rhel7.x86_64.rpm

And a new version of syslog-ng should be up and running with a fraction of a second downtime.

Steps will be similar with your own configuration, too. You might need a few more iterations, read the documentation, but as you can see, it is no more difficult than any other major software upgrade.