Creating your first block for the syslog-ng configuration library (SCL)

The syslog-ng configuration library (SCL) is a collection of ready-to-use configuration snippets that hide away the complexity of the specifics of your log processing pipeline. If you already reuse parts of your configuration on different machines, it is not rocket science to turn these parts into blocks, making your configuration files a lot simpler. And if they are not site-specific, you can even help other syslog-ng users by submitting them to be included in SCL, distributed in the official syslog-ng releases.

From this blog you can learn how to use inline configuration, how to create a block and how to test it. Finally, you can learn the steps of submitting your work for inclusion in syslog-ng. The example configuration is based on the Linux audit parsing I did in a previous blog. You can read that at https://www.syslog-ng.com/community/b/blog/posts/hook-commands-easy-driver-setup.

Original configuration

Here is the configuration we use as a starting point:

source s_auditd {
  file("/var/log/audit/audit.log" flags(no-parse)
    hook-commands(
      startup("auditctl -w /etc/ -p wa")
      shutdown("auditctl -W /etc/ -p wa")
    )
  );
};

parser p_auditd {
    linux-audit-parser (prefix("auditd."));
    kv-parser (template("${auditd.msg}") prefix("amsg."));
};

filter f_ssh {
  "${amsg.terminal}" eq "ssh"
};

filter f_path {
  "${auditd.type}" eq "PATH"
};

filter f_ssh_or_path {
  filter(f_ssh) or filter(f_path)
};

destination d_elastic {
  elasticsearch2 (
    cluster("syslog-ng")
    client_mode("http")
    index("syslog-ng")
    type("test")
    template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")
  )
};

log {
  source(s_auditd);
  parser(p_auditd);
  filter(f_ssh_or_path);
  destination(d_elastic);
};

You can read the abovementioned blog if you want to learn the motivation and implementation details behind the configuration.

Why to use a block / SCL?

Sometimes it is not worth reusing parts of your syslog-ng configuration, because it is too specific to a single server. Using blocks are also not necessary when you are using one of the many configuration management implementations (Puppet, Ansible and so on) to distribute the same configuration to a large cluster of machines.

However, most of the time you can find large and complex parts of your configuration that you can reuse with slight modifications on multiple machines. This is when turning the core of this configuration into a block comes handy. It can hide away the complex part of your configuration and can provide some sensible defaults that you can easily override when you really need it.

If you look at the configuration above, you can see a file source controlling and reading Linux audit logs. The collected logs are parsed, filtered and sent off to Elasticsearch for storage and analysis.

When collecting audit logs, the specifics of fetching data and doing an initial round of parsing stays the same across multiple machines. What you do as post-processing and how you store messages is what changes between systems.

When you put the opening and parsing parts into a separate block, the rest of your configuration is a lot easier to understand and maintain. And if you contribute your block to the SCL – like I did – you can start concentrating on your site-specific configuration from the next release.

Inline configuration objects

A lesser known feature of the syslog-ng configuration is that starting with version 3.4 you can use configuration objects inline. It often looks confusing on first use, because we are used to having all objects defined separately and connected by using their names in log statements. However, if we use an object only once, it is faster to write and easier to read inline objects.

For example, this configuration part that was taken from the documentation:

source s_local {
    system();
    internal();
};
destination d_local {
    file("/var/log/messages");
};
log {
    source(s_local);
    destination(d_local);
};

Can easily be turned into:

log {
    source {
        system();
        internal();
    };
    destination {
        file("/var/log/messages");
    };
};

Note that because we only used the s_local source once, we simply inlined that into the log statement. We did the same with the file() destination.

Among many other advantages, this way you do not have to use names for objects. This is also what I have used when writing the block for the Linux audit source in the first configuration example.

Using channels

The use of inline objects makes it possible to define source drivers within our blocks without having to give them a name (for example, s_local or similar). This name could also collide with user configuration.

But our goal this time is to define a new source that bundles reading the audit.log file together with the follow-up processing of messages, for example, parsing the audit log format. This bundling is possible by using channels.

The resulting configuration looks like the following:

source linux-audit {
  channel {
    source { file("/var/log/audit/audit.log" flags(no-parse)); };
    parser { linux-audit-parser (prefix(".auditd.")); };
    parser { kv-parser (template("${.auditd.msg}") prefix(".auditd.msg.")); };
    rewrite { unset(value(".auditd.msg")); };
  };
};

This configuration is almost finished:

• It is a proper syslog-ng source definition.
• It instructs syslog-ng to read the audit.log file.
• It also extracts key-value pairs from the log messages.

What it lacks at this point is proper reusability and flexibility. The file name and the prefix of name-value pairs are fixed. Also, you cannot add hook-commands() when calling this source. We will add these in the Creating a block section.

Creating a block

Using a block statement, we can turn the above source element into a more flexible Linux audit source:

block source linux-audit(filename("/var/log/audit/audit.log") prefix(".auditd.") ...) {
  channel {
    source { file("`filename`" flags(no-parse) `__VARARGS__`); };
    parser { linux-audit-parser (prefix("`prefix`")); };
    parser { kv-parser (template("${`prefix`msg}") prefix("`prefix`msg.")); };
    rewrite { unset(value("`prefix`msg")); };
  };
};

Here, the configuration starts with block. The rest looks mostly similar to the previous version, but there are some notable changes.

• Instead of a source {} statement to be referenced from log {} statements, this defines a source driver, that can actually be used in a source {} statement.
• The name of the source is followed by a list of possible parameters, coming with default values, for example, prefix(".auditd."). You can override the values if necessary when you refer to the block.
• The three dots (…) refer to any additional parameters. The extra parameters are inserted where `__VARARGS__` is found in the block.
• You can refer to the parameter values by enclosing the parameter name between apostrophes (``). For example, in the previous example, we refer to the value of the filename() parameter ("/var/log/audit/audit.log") in file("`filename`").

Testing the block

You can use the configuration below to test your new block:

source s_auditd {
  linux-audit(
    prefix("test.")
    hook-commands(
      startup("auditctl -w /etc/ -p wa")
      shutdown("auditctl -W /etc/ -p wa")
    )
  );
};

destination d_elastic {
  elasticsearch2 (
    cluster("syslog-ng")
    client_mode("http")
    index("syslog-ng")
    type("test")
    template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")
  )
};

destination d_json {
  file("/var/log/json" template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)\n\n"));
};

log {
  source(s_auditd);
  destination(d_elastic);
  destination(d_json);
};

Our block has been used as a driver in the s_auditd source {} statement, just as if it was a native driver implemented by syslog-ng.

To test the configuration, use either the JSON-formatted file destination and/or the Elasticsearch destination.

Also, if you take a detailed look at the s_auditd source you can see that when using the new linux-audit() source, it overrides the name-value prefix with test. and also adds hook-commands() to enable and disable auditing. The hook-commands() is not a pre-defined parameter of the block, but made possible by using … as demonstrated in the Creating a block section. The `__VARARGS__` part makes sure that it is used in the right place. Change a password or create a new file under /etc/ and you will see that the associated audit logs are generated and delivered to /var/log/json or Elasticsearch.

Submitting your block to be part of SCL

If the block you created is not too specific to your environment, you should consider submitting it to syslog-ng. Submitting has many advantages both for the community and for you.

The community gains a new block in the SCL simplifying the use of syslog-ng even further.

You gain several more benefits:

• You do not have to maintain the configuration block yourself.
• Peer review for your block, you will probably even receive a few good enhancement ideas.
• Fix from developers if the syntax of syslog-ng changes for any reason.

The source code of syslog-ng is on GitHub. To contribute, you will need the following:

• A GitHub account.
• Git installed on your machine.

I will provide you a simplified overview of using Git and GitHub. If you are a Git guru, you can safely skip most of this content, except the requirements of writing a proper commit message that is described in the Commit the file locally section.

Fork and clone the source code

As a first step, login to your GitHub account. You can find the syslog-ng source code at https://github.com/balabit/syslog-ng. First, you will add the syslog-ng source code repository to your user account (“fork”, in Git terminology) and then download it (“clone”, in Git terminology), so that you can add your block to it.

• To create a copy of the syslog-ng source code repository under your user name, click Fork (in the upper right corner). My GitHub user name is “czanik”, so the URL of the new repository in my example is accessible at https://github.com/czanik/syslog-ng. Your URL will include your user name, respectively.
• Once the fork is ready, create a local clone of it. Open a terminal window and enter the clone command. In my example, I would run the following command in my terminal:
• git clone https://github.com/czanik/syslog-ng
  When you run this command, replace my user name with your own.

Add file to local repository

Look at the directory you have just cloned from GitHub. It has a sub-directory called scl. If you list or view the contents of that directory, you can see that there are several sub-directories inside, named after the block they contain and a similarly named file with a .conf extension in them. For example, the telegram directory contains a file called telegram.conf. Open the .conf file to see that it is not just the configuration, but it also contains some boilerplate text about usage and copyright. You will have to include this text in your .conf file too.

• Create a new directory under scl based on the name of the block you have created. In this case, linux-audit.
• Copy an existing configuration file there, for example telegram.conf.
• Rename this copied file to the name of your block. In this case, it is linux-audit.conf.
• Open the .conf file to edit it and replace the configuration part with the block you have just created. Make sure that the boilerplate text is intact, and that the year reflects the current actual date.
• Save the file.

Commit the file locally

If it is your first time using Git, start with a little setup. If you are already a Git user, you can safely skip the next two commands. These configure the name and e-mail address used in Git commits. Enter the two commands in a terminal subsequently. Use your user name and email address, respectively. For example,

git config --global user.name "Peter Czanik"
git config --global user.email peter@czanik.hu

Before you can commit the file to the repository, you have to add it to the repository. Using the directory and file name above, enter the following command in the local syslog-ng directory where you have previously cloned the syslog-ng Git repository:

git add scl/linux-audit/

Now you are ready to upload (“commit”, in Git terminology) the changes to the local repository. Enter the following command:

git commit

You are asked for a commit message. This should start with a commit title line that is basically a short description of what you are about to upload, followed by a longer description of the new feature (in our case, the new block) and preferably also an example on how it can be used. At the end of this description, use the “Signed-off-by:” line to add your name and e-mail address, so people know that you are the original author of the code.

Here is the commit message that I wrote about linux-audit.conf

    scl: new linux-audit() source
    
    Add a new SCL: linux-audit() source. It reads and automatically
    parses the Linux audit logs. You can override the file name using
    the filename() parameter and the prefix for the created
    name-value pairs using the prefix() parameter. Any additional
    parameters are passed to the file source.
    
    Example:
    
    source s_auditd {
      linux-audit(
        prefix("test.")
        hook-commands(
          startup("auditctl -w /etc/ -p wa")
          shutdown("auditctl -W /etc/ -p wa")
        )
      );
    };
    
    Signed-off-by: Peter Czanik <peter@czanik.hu>

For more details on proper commit message formatting check the contribution guide at https://github.com/syslog-ng/syslog-ng/blob/master/CONTRIBUTING.md

Create a pull request

When you submit code to be included in the official syslog-ng project code repository, it is called a pull request or PR in Git terminology. Before you can do that from the GitHubweb interface, there is one more command to run locally in your terminal. You need to upload (“push”, in Git terminology) your changes from the local repository on your computer to your repository on GitHub. To do this, enter the following in your terminal:

git push

You will be prompted for your GitHub user name and password. Once you have entered those credentials correctly, your changes are uploaded to GitHub.

To submit your code, open the GitHub web interface in a browser, login with your credentials and navigate to the forked syslog-ng repository.

Click New pull request (close to the upper left corner). You should see your freshly uploaded commit on screen. To submit your changes to the balabit/syslog-ng repository, click Create pull request. After this, watch your PR for updates. Your code will be tested automatically. Once those tests are ready, two of the syslog-ng maintainers review your code to make sure that it is up to the syslog-ng coding and formatting standards and recommendations. They might ask for changes before your code is accepted into the official syslog-ng source code.

Once your pull request is accepted, your changes are included in the source code and will be part of the next release.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/balabit/syslog-ng. On Twitter, I am available as @PCzanik.