Build your own syslog-ng packages from patched Git sources

When a user reports a problem with syslog-ng, developers create code to fix it. The fix is called a patch or a pull request (PR) in Git terminology. Often this code does not enter the main source code automatically but developers ask the user to test the fix before the code is merged. A new challenge starts here: figuring out how to test a proposed fix. My blog shows you how to download and patch the syslog-ng source code and build easy-to-install (and remove)  packages from it for CentOS 7 and a few Ubuntu and Debian releases.

Creating a generic developers’ guide is beyond the scope of this blog. Instead, I will show you how to check out the latest syslog-ng sources from GitHub, the server where the syslog-ng source code is maintained. It is easier to test fixes or new features if syslog-ng is available as a package. Luckily, syslog-ng provides a Docker-based build environment to create a source Tarball and binary packages for some of the most popular Linux distributions.

You will learn how to:

1. Prepare your environment for downloading syslog-ng sources.
2. Download syslog-ng sources and apply the proposed fix for your issue.
3. Create a source Tarball using the patched syslog-ng source.
4. Prepare the build image.
5. Build the packages.
6. Clean up your machine.

Preparing your environment

There are a few dependencies you need to install before you can download, patch, and package syslog-ng. Some of these are most likely already available on your machine. As package names vary from distribution to distribution, I just list the commands here that you need:

• wget
• patch
• git
• Docker

Note that installing Docker is not enough, you also need to start it.

You now have all software required to download syslog-ng sources and create packages from.

Downloading and patching syslog-ng sources

In this section, you will learn how to download the syslog-ng sources from Git, find a patch that you are interested in, download the patch, and then apply it to the syslog-ng source code. The patch I use in the example below is already merged (part of the syslog-ng code base), so use another pull request (PR). If you have an open issue and a developer asked you to test a PR, use that. Otherwise browse https://github.com/balabit/syslog-ng/pulls for open pull requests. Replace the ID number in the below example with the number of the PR you need or want to test.

1. Download syslog-ng sources using Git:

git clone https://github.com/balabit/syslog-ng

This will create a new sub-directory – called syslog-ng – in your current working directory and place the source code there.

2. Once the download is finished, change to this new sub-directory:

cd syslog-ng

3. When the developers provide a fix for your issue or create an enhancement based on your request, they will provide you with a URL pointing to this fix or enhancement, called a pull request (PR). The following is an example PR for an syslog-ng feature: https://github.com/balabit/syslog-ng/pull/1856

As you can see, a PR is not only a patch with a new feature, but more than that. A PR is more like a description and a discussion of the patch. If you want to download the patch, append .patch to the URL of the PR and your browser will be redirected to a patch file. For the previous example, the URL will look like the following: https://patch-diff.githubusercontent.com/raw/balabit/syslog-ng/pull/1856.patch

To download the patch from the command line, for example with wget, enter the following command:

wget https://patch-diff.githubusercontent.com/raw/balabit/syslog-ng/pull/1856.patch

4. Apply the patch (modify the syslog-ng source code using the patch you downloaded) using the patch utility:

patch -p1 < 1856.patch

Creating a source Tarball

Developers usually work on Git sources. These often require installing additional software and packages, and therefore these are more time-consuming or difficult to use. On the other hand, Linux distributions and end-users usually use something called the source Tarball. The source Tarball includes much more than a simple syslog-ng source, to make the life of end-users easier. Along with syslog-ng sources, it also contains sources for some of the dependencies, it has generated man pages (help documentation and user manual), a configure script that you can use to configure which syslog-ng features are enabled, and a few other things.

You might have wondered why you had to install Docker. There is a tool in the syslog-ng Git sources that utilizes Docker to make sure that source Tarball generation happens in a well-defined environment with all the required tools and dependencies available.

To build the source Tarball, you do not need to have in-depth Docker knowledge, because it only requires issuing a single command. Issue the following command from the syslog-ng source code directory:

dbld/rules tarball

You will find the generated source Tarball in the dbld/build directory. Right now (until the next syslog-ng release that will modify the version number), the name of the generated file is syslog-ng-3.16.1.tar.gz

Note: If your distribution of choice is not supported by the syslog-ng package building system or you have your own package building system, your journey stops here. Otherwise go on with the next step: preparing the build image.

Preparing the build image

Before you can build the syslog-ng package, you have to prepare a build image using Docker. As with building the source Tarball, you do not have to deal with Docker directly, but through the dbld/rules script.

As a first step, you have to choose your target distribution. If you run the dbld/rules script without any arguments, it shows you a basic help message, including the list of supported platforms. Note that CentOS 6 does not work, even if listed there.

For example, if you have CentOS 7 (or RHEL 7, fully compatible from a syslog-ng point of view), you can use the centos7 target. To prepare the build image, you use the same dbld/rules script as previously but this time using the image parameter. Just append a dash and the target name to it, like:

dbld/rules image-centos7

This command downloads a minimal distribution image and installs all software and dependencies necessary to build the syslog-ng package inside the image. Once you get back the command prompt, you are ready to build the packages.

Building the packages

The last step is building the packages. You prepared the build image previously, now you will start using it. The command to run is very similar to the previous one. Just replace “image” with “rpm” or “deb”, depending on the package format of the target distribution (yes – thanks to Docker – you can build syslog-ng for Debian on an openSUSE machine and also the other way around).

Here are two examples. The first for CentOS 7, the second for Ubuntu Xenial (assuming that you also prepared the build image previously):

dbld/rules image-centos7

dbld/rules deb-xenial

You will find the results – the freshly built packages – in the dbld/build sub-directory, ready to be installed by your package manager.

Cleaning up your machine

The build environment you prepared can consume considerable disk space. You should clean up your machine once the packages are ready.

Once you copied the packages from the dbld/build sub-directory, you can safely delete the syslog-ng source code.

When you list Docker images, you can see similar results on your screen:

czanik@linux-6chp:~/syslog-ng> docker images

REPOSITORY                  TAG                 IMAGE ID            CREATED             SIZE

balabit/syslog-ng-centos7   latest              d66711d77469        14 minutes ago      1.63GB

ubuntu                      16.04               e13f3d529b1a        8 days ago          115MB

[...]

centos                      7                   49f7960eb7e4        7 weeks ago         200MB

balabit/syslog-ng-xenial    latest              fbac3be5dc49        4 months ago        1.18GB

The ubuntu:16.04 and balabit/syslog-ng-xenial images are used for Tarball building, independent of your target distribution. In the above listing, centos:7 and balabit/syslog-ng-centos7 were used to build the CentOS 7 packages. You should see similar listings for other distributions as well.

What is next?

In an ideal case, you built the package successfully, installed the results, and reported back to syslog-ng developers that your issue has been solved. The fix will be part of the next release.

Are you stuck?

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/balabit/syslog-ng. On Twitter, I am available as @PCzanik.