Collecting process accounting logs on Linux with syslog-ng

Process accounting logs are collected into binary log files on Linux. You can turn them into human readable format locally, using various tools. You can also use syslog-ng to read those files. Syslog-ng can parse those binary logs, create name-value pairs from them and store the results.

Personally, I have never tried this source previously. First of all, I do not use process accounting. And even if the configure script of syslog-ng mentions the feature in its help text, it is marked experimental:

  --enable-pacct          Enable support for reading Process Accounting files (EXPERIMENTAL, Linux only).

Which means, that previously, I have never enabled this feature in any of my RPM packages. Recently, someone asked me to enable it in my unofficial SLES 12 packages. First, I compiled packages for testing. When the user had problems (logs not showing up), I tried the pacct() source myself. This testing became the basis of my blog.

Before you begin

You need syslog-ng with Linux process accounting enabled. You can check if it is enabled in your current installation by checking if it appears in the output of “syslog-ng -V”:

czplaptop:~ # syslog-ng -V | grep pacct
Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,azure-auth-header,basicfuncs,cef,confgen,cryptofuncs,csvparser,dbparser,disk-buffer,examples,graphite,hook-commands,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,sdjournal,secure-logging,stardate,syslogformat,system-source,tags-parser,tfgetent,timestamp,xml,http,regexp-parser,pacctformat

There is a good chance that Debian and Ubuntu packages have it (not verified), either in the core package or one of the sub-packages. Until now, this feature has not been enabled in any of the RPM packages. I enabled it now in my git snapshot repositories, you can use those for testing.

You also need process accounting installed and started on your host. As the original request came from a SUSE user, here I show you the SUSE way, but it should be similar on other distributions as well:

zypper install acct
systemctl start acct

The pacct() source of syslog-ng expects process accounting logs to appear at /var/log/account/pacct. Check, if your system uses the same location, and use the file() parameter of the pacct() source to override the default location if necessary.

Configuring and testing syslog-ng

Normally, first I explain the configuration and then I show how to test it. Here I take a different route, as we will test multiple iterations of the configuration.

If your Linux distribution of choice supports it, create a new configuration under the /etc/syslog-ng/conf.d directory, otherwise append it to syslog-ng.conf

source s_pacct { pacct(); };
destination d_pacct {
  file("/var/log/frompacct");
};
log {source(s_pacct); destination(d_pacct);};

This simple configuration uses the pacct() source to collect process account logs, and results are written directly to a text file.

Reload the configuration to make it live and check the content of /var/log/frompacct to see your process accounting logs. Well, it is probably not what you expected:

Aug 27 09:15:27 czplaptop
Aug 27 09:15:27 czplaptop
Aug 27 09:15:27 czplaptop

That is right, you only see a message header, but the message part is empty. It is because process accounting logs are not log messages in a traditional syslog sense, but a collection of key=value pairs and syslog-ng extracts those into name-value pairs. When it comes to structured logs, I tend to use JSON formatting for an initial look, especially that in the end, those logs often end up in Elasticsearch for storage and visualization. Let us try a different configuration:

source s_pacct { pacct(); };
destination d_pacct {
  file("/var/log/frompacct"
    template("$(format-json --scope rfc5424 --scope dot_nv_pairs)\n")
  );
};
log {source(s_pacct); destination(d_pacct);};

This configuration adds JSON formatting to the log file. It uses the format-json template function and logs both name-value pairs from a syslog header and name-value pairs parsed from the log message. We use dot_nv_pairs as a scope, as the pacct() source stores parsed values with a .pacct prefix.

{"_pacct":{"ac_utime":"0.00","ac_uid":"0","ac_tty":"34816","ac_swaps":"0","ac_stime":"0.00","ac_rw":"0","ac_ppid":"6848","ac_pid":"8455","ac_minflt":"115","ac_mem":"21016","ac_majflt":"0","ac_io":"0","ac_gid":"0","ac_flag":"01","ac_exitcode":"0","ac_etime":"     0.00","ac_comm":"bash","ac_btime":"21254144.00"},"PRIORITY":"notice","HOST":"czplaptop","FACILITY":"user","DATE":"Aug 27 09:29:10"}
{"_pacct":{"ac_utime":"0.00","ac_uid":"0","ac_tty":"34816","ac_swaps":"0","ac_stime":"0.00","ac_rw":"0","ac_ppid":"6848","ac_pid":"8456","ac_minflt":"138","ac_mem":"21016","ac_majflt":"0","ac_io":"0","ac_gid":"0","ac_flag":"01","ac_exitcode":"0","ac_etime":"     0.00","ac_comm":"bash","ac_btime":"21258240.00"},"PRIORITY":"notice","HOST":"czplaptop","FACILITY":"user","DATE":"Aug 27 09:29:10"}

As you can see, log messages are not empty anymore. JSON is good for debugging and Elasticsearch also, but you might want to write some more human readable logs in the end. Here is another template for that:

source s_pacct { pacct(); };
destination d_pacct {
  file("/var/log/frompacct"
    template("${DATE} ${HOST} pacct $(format-welf --scope dot_nv_pairs)\n")
  );
};
log {source(s_pacct); destination(d_pacct);};

This way the created logs are similar to regular syslog messages. Firewalls, sudo, and many others use a similar format:

Aug 27 09:53:44 czplaptop pacct .pacct.ac_btime=27291648.00 .pacct.ac_comm=cat .pacct.ac_etime="     0.00" .pacct.ac_exitcode=0 .pacct.ac_flag=00 .pacct.ac_gid=0 .pacct.ac_io=0 .pacct.ac_majflt=0 .pacct.ac_mem=4796 .pacct.ac_minflt=89 .pacct.ac_pid=9496 .pacct.ac_ppid=6848 .pacct.ac_rw=0 .pacct.ac_stime=0.00 .pacct.ac_swaps=0 .pacct.ac_tty=34816 .pacct.ac_uid=0 .pacct.ac_utime=0.00
Aug 27 09:53:44 czplaptop pacct .pacct.ac_btime=27291648.00 .pacct.ac_comm=bash .pacct.ac_etime="     0.00" .pacct.ac_exitcode=0 .pacct.ac_flag=01 .pacct.ac_gid=0 .pacct.ac_io=0 .pacct.ac_majflt=0 .pacct.ac_mem=21016 .pacct.ac_minflt=113 .pacct.ac_pid=9498 .pacct.ac_ppid=9497 .pacct.ac_rw=0 .pacct.ac_stime=0.00 .pacct.ac_swaps=0 .pacct.ac_tty=34816 .pacct.ac_uid=0 .pacct.ac_utime=0.00

What is next?

As you could see from the examples, there are endless possibilities for formatting the output. There are three major topics in the documentation that help you to better understand formatting:

• macros and name-value pairs are the content: values parsed from log messages by syslog-ng

• template functions are the magic ingredient, creating JSON and other formatting

• templates are where you combine macros, free text and template functions together to create your output

-

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.