This is the final blog post in a three-part series on logging in Docker using syslog-ng. I have already covered how to use syslog-ng in a Docker environment as a traditional central syslog server and how to collect host and container logs from the host journal. There are many software that log to files or pipes instead of their stdout, the place where Docker expects them. Fortunately, by using Docker volumes, you can share data among containers and syslog-ng can collect these logs as well.
Before you begin
In this blog, I detail methods of collecting logs from other containers that are not covered by Docker’s own log collecting method (collecting everything sent to the stdout of a container as log). For other use cases, read my previous blogs in the series:
- Running your central syslog-ng server in Docker is the most traditional use case: sending logs over the network to syslog-ng running in a container.
- Collecting Docker infrastructure logs describes how to collect Docker logs (both host logs and container stdout) using syslog-ng running in a container and reading the hosts journal.
You can also read the whole Docker series in a single white paper.
To make the most of this blog post, you should be familiar with how Docker handles volumes. In my examples, I use the legacy “-v” syntax and use mostly bind mounts. This syntax works with older Docker releases as well. It should not be difficult to turn those to the “–mount” syntax and use volumes instead in most cases. If these are new to you, read the Docker storage documentation at https://docs.docker.com/engine/admin/volumes/.
Reading logs from files
This time you will read log files from other containers using Docker volumes. There will be a container running syslog-ng and two other containers sending log messages. This example is just a laboratory experiment but with minimal modifications you should be able to use it to collect messages from web servers with many virtual hosts logging to separate files.
You will need four terminal windows:
- Terminal 1 and Terminal 2: In the first two you will start up Alpine Linux in interactive mode. I use Alpine Linux Docker images in my examples as it is the quickest to download and start due to its extreme small size. You can use your own favorite Linux distribution with minor modifications of the command line if you really want to.
- Terminal 3: In the third window you start up syslog-ng.
- Terminal 4: In the fourth one you can check your logs.
Preparations
First, start up two containers in two separate terminal windows. You will use them later on to create some log messages.
Steps:
Step 1.
Go to Terminal 1 and enter the following command:
docker run -ti -v container1:/var/log alpine /bin/ash
This command will download an Alpine Linux image and start it in interactive mode due to the “-ti” option. Alpine uses “ash” as shell, so this example uses it as a command to start. The most important part of the command line is “-v container1:/var/log”. It automatically creates a Docker volume, called “container1” (if it does not exist yet) and makes it available in the container under the /var/log directory.
Step 2.
Go to Terminal 2 and enter the following command:
docker run -ti -v container2:/var/log alpine /bin/ash
Step 3.
Go to Terminal 3. In this window you will prepare your syslog-ng container. Here, you will use bind mounts as this way files are easier to edit and check from the host machine.
Step 4.
Create a directory structure under the /data/syslog-ng directory for the configuration and logs.
mkdir -p /data/syslog-ng/conf /data/syslog-ng/logs
Step 5.
Create a new syslog-ng configuration file and save it as /data/syslog-ng/conf/filread.conf with the following content:
@version: 3.11
source s_internal {
internal();
};
destination d_int {
file("/var/log/int");
};
log {source(s_internal); destination(d_int); };
source s_wild { wildcard-file(
base-dir("/var/log/common")
filename-pattern("*.log")
recursive(yes)
follow-freq(1)
);
};
destination d_fromwild {
file("/var/log/fromwild"
template("$(format_json --scope rfc5424 --scope nv_pairs)\n\n")
);
};
log {source(s_wild); destination(d_fromwild);};
The first three lines ensure that the internal messages from syslog-ng are saved to a file. The “s_wild” source defines a wildcard file source. You will see later on that log directories from other containers are mapped in sub-directories under the /var/log/common directory of the syslog-ng container. Files matching the “*.log” pattern are read by syslog-ng in any of the sub-directories recursively under /var/log/common. The “d_fromwild” destination saves logs to a file with JSON formatting, so you will be able to see file names in the file too.
Step 7.
Finally, start syslog-ng using the following command line:
docker run -ti -v container1:/var/log/common/c1 -v container2:/var/log/common/c2 -v /data/syslog-ng/conf/fileread.conf:/etc/syslog-ng/syslog-ng.conf -v /data/syslog-ng/logs/:/var/log --name sng_fileread balabit/syslog-ng:latest
This starts the latest Balabit syslog-ng image where the options mean:
- “-ti” interactive mode, makes testing and debugging easier. Use “-d” in a production environment.
- “-v container1:/var/log/common/c1 -v container2:/var/log/common/c2” maps the Docker volumes for log directories from the two other containers to sub-directories under /var/log/common directory.
- “-v /data/syslog-ng/conf/fileread.conf:/etc/syslog-ng/syslog-ng.conf -v /data/syslog-ng/logs/:/var/log” maps the configuration file and log directories from the host file system.
- “–name sng_fileread” names the container.
- “balabit/syslog-ng:latest” starts (and downloads – if necessary) the latest Balabit syslog-ng image from the Docker hub.
Testing
This test will send a sample log message from the first and the second container to syslog-ng that you will be able to check in your fourth terminal window.
Steps:
Step 1.
Go to Terminal 1 and enter:
echo "from container1" >> /var/log/first.log
Step 2.
Go to Terminal 2 and enter:
echo "from container2" >> /var/log/second.log
Expected outcome:
Go to Terminal 4 and check if logs arrive. If you have configured everything correctly, you should see something like the following examples:
# tail -f /data/syslog-ng/logs/fromwild
{"SOURCE":"s_wild","PROGRAM":"from","PRIORITY":"notice","MESSAGE":"container1","LEGACY_MSGHDR":"from ","HOST_FROM":"525628e0a8e5","HOST":"525628e0a8e5","FILE_NAME":"/var/log/common/c1/first.log","FACILITY":"user","DATE":"Sep 1 08:04:51"}
{"SOURCE":"s_wild","PROGRAM":"from","PRIORITY":"notice","MESSAGE":"container2","LEGACY_MSGHDR":"from ","HOST_FROM":"525628e0a8e5","HOST":"525628e0a8e5","FILE_NAME":"/var/log/common/c2/second.log","FACILITY":"user","DATE":"Sep 1 08:05:14"}
Reading logs from pipes
If you want to avoid dealing with log files, allocating disk space, rotating them, and so on, you can often use pipes instead. For simplicity you will reuse the previous test environment.
Preparation
You will need the four terminal windows that you have prepared in the Reading logs from files section.
Steps:
Step 1.
Go to Terminal 3. Append the following lines to /data/syslog-ng/conf/fileread.conf:
source s_pipe { pipe("/var/log/common/c1/pipe");};
destination d_frompipe {file("/var/log/frompipe");};
log {source(s_pipe); destination(d_frompipe);};
The pipe source automatically creates the pipe if it is not yet already there. In this case the pipe will also show up in the /var/log directory of the first container.
Step 2.
Restart your syslog-ng container:
Step 2/a.
To stop the container, press Ctrl-C.
Step 2/b.
To start the container again, enter:
docker start -a sng_fileread
Note that this command will only work if you have used the previous test environment from the Reading logs from files section. If you have named the container something else than “sng fileread” (in Step 7. in Preparations), make sure to use that name instead.
This starts up the container named “sng_fileread” again. The “-a” option makes sure that it runs in the foreground where you can immediately see any error messages from syslog-ng.
Testing
This test will send data into the pipe and check whether it arrives.
Steps:
Step 1.
Go to Terminal 1.
Step 2.
To make sure that the pipe is there, enter
ls -l /var/log
It will display something like the following:
total 4 -rw-r--r-- 1 root root 16 Sep 1 08:04 first.log prw------- 1 root root 0 Sep 1 09:21 pipe
Step 3.
Send some data into the pipe:
echo testing the pipe > /var/log/pipe
Expected outcome:
You should see the log message in /data/syslog-ng/logs/frompipe:
[root@localhost ~]# tail /data/syslog-ng/logs/frompipe Sep 1 09:57:04 650e03ee2170 testing the pipe [root@localhost ~]#
What is next?
In this blog I have shown you the basics of using Docker volumes to share logs among containers. I kept the examples simple and focused on testing instead of production usage. In real life you will most likely combine what you learned here with possibilities detailed in my previous two Docker blogs:
As usual, I omitted many details to keep my blog at a reasonable length. Here I list a few resources worth reading if you want to learn more or if you get stuck along the way:
- The official syslog-ng documentation: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide
- Docker storage documentation: https://docs.docker.com/engine/admin/volumes/
Read the entire series about logging in Docker using syslog-ng in a single white paper.
