Your favorite Linux distribution is X. You test everything there. However, your colleagues use distro Y, and another team distro Z. Nightmares start here: the same commands install a different set of syslog-ng features, configuration defaults and use different object names in the default configuration. I ran into these problems while working with Gábor Samu on his HPC logging blog.
From this blog you can learn about some of the main differences in packaging and configuration of syslog-ng in various Linux distributions and FreeBSD, and how to recognize these when configuring syslog-ng on a different platform.
Packaging
You will notice major differences already when you install syslog-ng. When you install the syslog-ng package on openSUSE / SLES or Fedora / RHEL (and compatibles), it installs a basic syslog-ng package with most files from SCL (syslog-ng configuration library). However, all syslog-ng modules that need extra dependencies are packaged separately to keep the footprint of the package minimal.
The syslog-ng package in FreeBSD follows a similar policy. The syslog-ng package only has features which do not require extra dependencies. The only exception is the http() destination, which is built-in, and needs curl. If you want any other features with extra dependencies, like Python or Riemann support, you must compile syslog-ng yourself from FreeBSD ports.
Packaging on Debian /Ubuntu is completely different. The syslog-ng package is an umbrella package, pulling in all syslog-ng sub packages with their dependencies. Hundreds of megabytes. On the other hand, if you install the syslog-ng-core package, then it is unusable by itself as SCL is missing, just as most modules, including those without extra dependencies. You really have to know what you are doing.
No matter which Linux distribution you are using, you will be surprised because a completely different set of syslog-ng features and dependencies will be installed from what is expected. One of the returning problems is that a driver implemented in SCL, like elasticsearch-http() expects the syslog-ng http module to be installed. Luckily, they now show a meaningful error message, including suggestions on which package(s) to install:
[root@rawhideserv conf.d]# syslog-ng -s Cannot load required module; module='http', details='The elasticsearch-http() driver depends on the syslog-ng http module, please install the syslog-ng-mod-http (Debian & derivatives) or the syslog-ng-http (RHEL & co) package', location='/usr/share/syslog-ng/include/scl/elasticsearch/elastic-http.conf:39:2' […]
In other cases, you have to figure out yourself, that if your configuration does not work if a Riemann destination is added, then you have to install the related syslog-ng module (if available).
Object names
Here is a (simplified) syslog-ng configuration:
source s_src {
system();
internal();
};
destination d_messages { file("/var/log/messages"); };
filter f_messages { level(info,notice,warn) and
not facility(auth,authpriv,cron,daemon,mail,news); };
log { source(s_src); filter(f_messages); destination(d_messages); };
Each object (source, filter, and so on) in the configuration has a name. This is how we can refer to various objects in the log path, where we connect these objects. The above configuration is from Debian, and the name for local log sources is called s_src. However, these names are not standardized. For example, in openSUSE / SLES the name for local logs is called src.
Why is this a problem? For example, let's say that you find a configuration snippet in a blog which addresses a problem that you are trying to solve. You copy & paste the snippet into a new configuration file in the directory /etc/syslog-ng/conf.d/. However, you happen to run SLES, but the config sample was made on a Debian system:
destination d_myfile {
file("/var/log/myfile");
};
log {
source(s_src);
destination(d_myfile);
};
You even remember to run a syntax check, and it does not show any problems:
czplaptop:/etc/syslog-ng/conf.d # syslog-ng -s czplaptop:/etc/syslog-ng/conf.d #
However, when you try to start syslog-ng, it shows an error:
czplaptop:/etc/syslog-ng/conf.d # syslog-ng [2024-04-17T09:23:38.613430] Error resolving reference; content='source', name='s_src', location='/etc/syslog-ng/conf.d/fail.conf:5:3'
If you have a recent enough syslog-ng version, you can do a full configuration initialization instead of just a syntax check:
czplaptop:/etc/syslog-ng/conf.d # syslog-ng --check-startup Error resolving reference; content='source', name='s_src', location='/etc/syslog-ng/conf.d/fail.conf:5:3'
For more details check https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-can-now-do-a-full-configuration-check
Defaults
This is where the “fun” begins. Well, kind of, as long as you like detective stories. The previously mentioned problems are annoying, but you get a straightforward error message. You can configure global options for syslog-ng in the options section of the configuration. This is yet another case where each Linux distribution or BSD variant has different settings. As there are no error messages, you must read the documentation to resolve these problems.
We initially started to work on RHEL, then switched to Debian. Suddenly, instead of host names we started to see IP addresses in log messages. In this case it was relatively easy. The syslog-ng.conf on RHEL has keep_hostname(yes) in options, while the one in Debian does not have this setting. The default value of this option is no, which means that syslog-ng replaced host names with the sender IP address. Adding this option to the configuration resolved the problem. Other problems might require a bit more time to recognize and more effort to resolve.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.