syslog-ng FAQ

Summary

This article describes how to help syslog-ng developers to diagnose and fix syslog-ng crash problems.

Compile with debug symbols

If you are compiling syslog-ng from source, include debug symbols using the '''--enable-debug''' configuration option or by passing the '''-g''' flag in the '''CFLAGS''' and '''LDFLAGS''' environment variables. This will produce a slightly larger syslog-ng binary which contains debug symbols. Dependencies should also be compiled with debug symbols. For most distributions debug information is contained in an additional package with a -dbg (Debian),-dbgsym (ubuntu) -debuginfo (Fedora/openSUSE) suffix.

Generating a core file

When syslog-ng crashes and the ulimit -c resource limit is set to unlimited or a large enough value, the operating system produces a core file upon crash. The location of the core file depends on how syslog-ng was started:

• if started in the background, then the location of the PID file (usually /var/run/syslog-ng for OSE or /opt/syslog-ng/var for PE)
• if started in the foreground, then the current directory

Make sure that the syslog-ng process has appropriate write permissions in this directory.

The name of the core file is somewhat OS dependent, it is usually simply named "core", but variations that include the pid number also exist.

Generating a backtrace

Once you have a binary that has debug symbols and a core file containing the memory map of the crash, you can generate a backtrace using gdb (the GNU Debugger) like this

$ gdb /sbin/syslog-ng -c core
(gdb) bt full
... ... backtrace output ...

Post the backtrace to syslog-ng GitHub issues at https://github.com/balabit/syslog-ng/issues as an attachment, or send it to the syslog-ng mailing list. The developers of syslog-ng might request additional information; sometimes the complete core file is required to diagnose the problem.

Additional information

• GDB documentation: http://sourceware.org/gdb/documentation/
• Wikipedia page on core dumps: http://en.wikipedia.org/wiki/Core_dump

Summary

This article describes, how syslog-ng can be used in a chroot environment

Running syslog-ng chrooted

Start syslog-ng using the ''--chroot'' command-line parameter. For example: '''syslog-ng --chroot /var/chroot'''.

before v3.0

The syslog-ng application initializes its configuration before the changing the root (e.g., the local UNIX domain socket ''/dev/log'' is opened before chrooting).

Note that it is not possible to reload the syslog-ng configuration after chrooting, thus you will need to use restart, or bind-mount the ''/dev'' directory into the chroot.

from v3.0

The syslog-ng application first changes the root and initializes its configuration file and opens files only afterward.

Read logs from chroot environment

One can add additional log sources to the configuration of the syslog-ng application running outside of the chroot, which point to /dev/log inside the chroot.

source local {
unix-stream("/dev/log");
internal();
};
source jail1 { unix-stream("/jail/dns/dev/log"); };
source jail2 { unix-stream("/jail/www/dev/log"); };

You can do this by using a single source:

source local {
unix-stream("/dev/log");
internal();
unix-stream("/jail/dns/dev/log");
};

Note that postfix appears to need a log socket in it's chroot jail, or it's logging will stop when you reload syslog-ng: source postfix { unix-stream("/var/spool/postfix/dev/log" keep-alive(yes)); };

Yes. Use a source line like this in your conf file to read kernel messages, too.

source src { file("/proc/kmsg"); unix-stream("/dev/log"); internal(); };

Or use the system() source introduced in recent versions of syslog-ng, which automatically configures all local, platform specific sources on a system.

Warning

There is problem caused by the ''/proc/kmsg'' implementation of the Linux kernel, namely, when two parallel processes open and try to read from the ''/proc/kmsg'' file.

This special file is the userspace interface for collecting kernel log messages. If there are multiple processes reading ''/proc/kmsg'', a race condition occurs and the process which loses the race essentially deadlocks until the next kernel message is generated, when a new race condition occurs.

If the system logger is blocked and is not processing userspace messages, slowly all system daemons stall while waiting for the system logger to become available again. This includes crucial system components like '''sshd''' or the application handling system logins.

There are two alternative solutions for processing kernel messages on Linux systems:

• using '''klogd''' from the sysklogd package
• reading ''/proc/kmsg'' from syslog-ng natively, without using the '''klogd''' daemon

The second method is recommended.

The solution to this problem is to avoid running multiple userspace components that touch ''/proc/kmsg''. If you configure syslog-ng to read kernel messages, ensure that '''klogd''' is not running. If you want to run '''klogd''', ensure that '''syslog-ng''' does not read ''/proc/kmsg''.

Status

syslog-ng OSE 2.0.7 & syslog-ng PE 2.1.9 includes a workaround for this problem, which wakes up syslog-ng after 10 seconds of blockage. This still means that the system is essentially stalled for 10 seconds every time syslog-ng loses this race, however, the system does not need to be power-cycled to resolve the situation.

Issue

The clocks of my syslog-ng clients are not synchronized properly. This causes problems when creating files using the time macros.

Solutions

As option

Using the '''keep_timestamp(no)''' option, syslog-ng uses the time of reception.

In templates

Use the date/time related macros with an "R_" prefix, like "R_DATE".

Issue

When using the ''program()'' destination driver, the script processing the messages exits after processing the first message.

Solution

Read log messages from stdin. Instead of fetching a single line and exiting, keep reading the input. For example in Perl this can be accomplished as follows:

#!/usr/bin/perl
while (<>) {
process messages }

Additional information

Please note, that syslog-ng automatically restarts applications started by ''program()'', which can cause unnecessary load on the machine if happens too often.

This depends on the configuration of syslog-ng, the hardware running syslog-ng, and the exact situation and environment. The limits of the throughput of syslog-ng are similar to the limits of other network applications. When sizing the hardware required for syslog-ng, try to estimate the log traffic to be processed and the parsing and filtering complexity to be applied.

The main limiting factors are the following:

• Network bandwith
• Disk I/O
• CPU (only in you have complex parsing and filtering setup)

Other limiting factors are:

• SSL, as it is very CPU intensive
• database destinations, as due to their complexity they can handle a lot less traffic than plain text files

Additional information

For more details, check syslog-ng pe v7.0 performance whitepaper!

(Obsolete, / and /usr is merged in distributions)

Issue

Syslog-ng 3.0 and later requires libraries which usually reside in ''/usr'', thus cannot be used on systems where ''/usr'' is mounted during boot time, after syslog-ng starts.

Solution

syslog-ng 3.0 and later can link to ''GLib'', ''EventLog'' and other libraries statically, thus it will not require those libraries to be present at boot time. This can be achieved by the '''--mixed-linking''' ''configure'' option.

Glad you asked, the most common side effect is being happy with a superior syslog daemon :-)

Another common result if you compiled syslog-ng yourself is that system logfiles grow to huge sizes. This isn't syslog-ng's fault, but a side effect of syslog-ng logging to different logfiles than your old syslog daemon. Change your log rotation program's (for example: logrotate) config files to rotate the new log names/locations or change syslog-ng's config file to make it log to the same files as your old syslog daemon. If syslog-ng was installed from a package for your Linux distribution, log rotation is already configured. You should still take care for any new log destinations you create.

An alternative is to use timestamp in log file names, effectively causing log rotation to be performed automatically by syslog-ng. One still needs to take care deleting old logs. The following simple commands achieve this, which are easy to customize:

 # Find all non-Archived files that aren't from today, and archive them
# Archive Logs are deleted after 14 days
#
#Changes. Change -mtime +14 to the number of days to keep
# Archive old logs
/usr/bin/find /var/log/HOSTS ! -name "*.gz" -type f ! -path "*`/bin/date +%Y/%m/%d`*" -exec /usr/bin/gzip {}\;
# Delete old archives
find /var/log/HOSTS/ -daystart -mtime +14 -type f -exec rm {}\;
# Delete empty directories
find /var/log/HOSTS/ -depth -type d -empty -exec rmdir {}\;

Yes, this is possible, if you can get the packaging system of your Linux/UNIX operating system to install both syslog-ng and syslogd at the same time. The syslog-ng application can be configured not to process locally generated messages, and local syslogd can be configured to send all log messages to syslog-ng using the UDP protocol using the following configuration line:

 *.* @127.0.0.1

Please note that syslogd might open the syslog UDP port automatically, in which case the above configuration will cause all syslog messages to be resent to itself, causing an endless loop.

The syslog-ng application can accept messages from syslogd using the ''udp()'' source.

One can also use a named pipe to send logs to syslog-ng. From man syslogd:

 This version of syslogd(8) has support for logging output to named pipes (fifos). A fifo or named pipe can be used as a
destination for log messages by prepending a pipe symbol ("|") to the name of the file. This is handy for debugging.
Note that the fifo must be created with the mkfifo(1) command before syslogd(8) is started.

Filters are optional. A catchall should appear in your '''syslog-ng.conf''' before all other entries; and can look something like:

 destination catchall {
file(/var/log/catchall);
};
log {
source(syslog);
destination(catchall);
};

When a log statement includes multiple filter statements, syslog-ng sends a message to the destination only if all filters are true for the message. In other words, the filters are connected with the logical AND operator. In the following example, no message arrives to the destination, because the filters are exclusive (the hostname of a client cannot be example1 and example2 at the same time):

 filter demo_filter1 { host("example1"); };
filter demo_filter2 { host("example2"); };
log demo_filteredlog {
source(s1); source(s2);
filter(demo_filter1); filter(demo_filter2);
destination(d1); destination(d2); };

To select the messages that come from either host example1 or example2, use a single filter expression:

 filter demo_filter { host("example1") or host("example2"); };
log demo_filteredlog {
source(s1); source(s2);
filter(demo_filter);
destination(d1); destination(d2); };

For more details, please check the documentation.

Question

Can I set the owner or permission of the created files?

Answer

Yes, you can do this using the ''owner()'', ''group()'' and ''perm()'' options. For example:

destination d_file { file("/var/log/$HOST/log" create_dirs(yes)
-> dir_owner("log") dir_group("log") dir_perm(0600) <-
owner("log") group("log") perm(0600)); };

It can be achieved by ''flush_lines(1)'', which is the default setting.

 options { flush_lines(1); };

It's on the todo list. As long as it is not implemented, you might try to redirect the program's output to a named pipe like this:

 destination d_swatch { program("swatch 2> /var/run/swatch.err"); };
source s_swatch { pipe("/var/run/swatch.err"); };

Yes, by default syslog-ng has a supervisor process, which monitors the child. If the child crashes, the supervisor process automatically restarts it. Its behavior is controlled by the command line: '''--process-mode=<foreground|background|safe-background>''' The default is '''safe-background''' which enables the supervisor. The other two disables it. In practice the child is the main process, the supervisor is only there to restart it in the following cases:

• it was killed by a signal
• it exited with a non-zero return value

When shutting down syslog-ng, the child process needs to receive a TERM signal, which will exit with a zero return value and also brings away the supervisor process.

Question

Can I use AppArmor with syslog-ng?

Answer

Yes, with some limitations. Those Linux distributions, which have AppArmor enabled, have also a profile for syslog-ng, which covers the syslog-ng version and configuration included in the distribution. What it means in practice, that syslog-ng installed from distribution sources and using default configuration works fine, and enjoys AppArmor protection. As soon as one starts to change syslog-ng.conf it is easy to run into problems. AppArmor protection is based on directory and file names. If syslog-ng is not installed to the distribution default location, usually ''/sbin/syslog-ng'', then syslog-ng is not protected / limited by AppArmor.

Modifying AppArmor settings

AppArmor settings for different applications are stored in so called ''profiles''. These can usually be found in the directory called ''/etc/apparmor.d/''. Profile names are derived from from path and application names, just slashes are replaced by dots. So for syslog-ng it is usually ''sbin.syslog-ng''. It has usually some comments and copyright info at top, then some includes, capabilities and then access control to files and directories. This is where additional entries should be added, if log files outside of /var/log should be read or written by syslog-ng. For example to write logs to ''/opt/myapp/logs/iwantlogshere.log'' add the following line: /opt/myapp/logs/iwantlogshere.log rw, Or if only read-only access is necessary, as the file is read by syslog-ng, processed and forwarded somewhere, then replace '''rw''' with '''r'''. Once a profile is modified, AppArmor should be restarted. There are some advanced syslog-ng features, for which it is quite difficult to add AppArmor protection. A prime example is the program() destination, where the called application either runs unprotected or a separate profile needs to be created. Calling external applications is tested to work with AppArmor v2.5 but did not work at all with version 2.3. See ''man apparmor.d'' for details on different execution permissions.

If you have AppArmor enabled, make sure that the given path is enabled in the AppArmor profile. Or – if you have SELinux – modify the SELinux rules.

It might be also a problem, that the directory does not exist. There are two possible solutions:

• create the directory by hand
• use the '''create_dirs(yes)''' option either globally, or locally for the specific file destination, so the necessary directories are created automatically