Why use syslog-ng for collecting Docker logs? Docker already provides many drivers for logging, even for central log collection. On the other hand, remote logging drivers arrive with a minimalist feature set and you are not able to use the “docker logs” command any more. To have the best of both worlds, you can use journald logging driver in Docker and use syslog-ng to read Docker logs from journald and forward log messages to your central log server or other destinations. You can even run syslog-ng itself in a Docker container, so you can use it on dedicated Docker host environments as well where it is not possible to install additional applications.
This is the second blog post in a three-part series on logging in Docker using syslog-ng. In the previous post, I talked about how to run your central log server in Docker. You’ll find a link to the next part in the series at the end of this post. You can also read the whole Docker series in a single white paper.
Why syslog-ng?
If you log in to one of your Docker hosts it is convenient to use the “docker logs” command to view logs of your containers locally. This only works if you use local logging, either the journald or the json-file logging method.
There are multiple logging drivers included in Docker for remote logging. Syslog and other protocols are supported. The problem with this approach is that these drivers are fairly limited. They include a minimalist feature set and it is not possible to filter or process log messages in any way before they are sent to their destination. Also, if the network connection to the central server is down for some reason, messages can easily be lost.
In an ideal world you can both use the “docker logs” commands locally and collect your infrastructure logs at a central location for long-time storage and analysis. This can be achieved by using journald driver in Docker for collecting logs. The journald driver is the default choice on Fedora Linux and derivatives like RHEL or CentOS. To enable it on openSUSE or SLES, add the –log-driver=journald option to the DOCKER_OPTS variable in /etc/sysconfig/docker and restart Docker.
You can use syslog-ng to read log messages from journald, process them, filter them and either store them locally, at a central syslog server or at one of the many supported destinations of syslog-ng, including Hadoop, Kafka, MongoDB or Elasticsearch.
The syslog-ng application can run directly on the host system but also in a Docker container. Running syslog-ng in a container is the only way if you use some of the dedicated Docker host environments – like Atomic Host – where you cannot even install applications directly on the host operating system.
I will show below how you can run syslog-ng in a container, collect structured log messages from journald and save them locally in JSON format preserve all data.
A simple configuration
First, create a simple configuration for syslog-ng. Once declaring the version number of syslog-ng, we also define two sources: one for the journal and another for syslog-ng’s internal messages. This helps resolving any run-time problems. Next, define two file destinations. For log messages coming from the journal, use a JSON template, so all name-value pairs can be recorded in a structured format. Docker creates some extra fields, like CONTAINER_NAME and different identifiers related to the containers. At the end the two log statements connect the previously listed building blocks together.
@version: 3.10 source s_journal { systemd-journal(prefix("journal.")); }; source s_internal { internal(); }; destination d_int { file("/var/log/int"); }; destination d_file { file("/var/log/journal.json" template("$(format_json --scope rfc5424 --key journal.*)\n\n")); }; log {source(s_journal); destination(d_file); }; log {source(s_internal); destination(d_int); };
This configuration is good enough to get you started and you can check that syslog-ng is already collecting log messages. Once everything works as expected, you can create more complex configurations with filters, Big Data destinations and more.
Starting syslog-ng
The following command starts syslog-ng in a Docker container. It assumes that the configuration and data are not in the container, but mapped from the host file system. This makes configuring syslog-ng and checking the log files easier. The path for the configuration is /data/syslog-ng/conf/journal.conf and for the logs it is /data/syslog-ng/logs/.
docker run -ti -v /etc/machine-id:/etc/machine-id -v /data/syslog-ng/conf/journal.conf:/etc/syslog-ng/syslog-ng.conf -v /data/syslog-ng/logs/:/var/log -v /var/log/journal:/var/log/journal --name journal balabit/syslog-ng:latest
When you start the container using the command above, it will be started in the foreground and you will not receive the command prompt back. While it makes debugging easier, it is not for production use. In my case I will see a message about configuration versions (running 3.10 with a 3.9 configuration), but in an ideal case nothing is displayed on screen:
[root@localhost ~]# docker run -ti -v /etc/machine-id:/etc/machine-id -v /data/syslog-ng/conf/journal.conf:/etc/syslog-ng/syslog-ng.conf -v /data/syslog-ng/logs/:/var/log -v /var/log/journal:/var/log/journal --name journal balabit/syslog-ng:latest syslog-ng: Error setting capabilities, capability management disabled; error='Operation not permitted' [2017-06-27T09:29:19.032130] WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode Please update it to use the syslog-ng 3.10 format at your time of convenience, compatibility mode can operate less efficiently in some cases. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file.;
Let’s see the parameters of this Docker command in detail:
- “run” starts a command in a new container
- “-ti” starts interactive mode (use -d in production)
- “-v /etc/machine-id:/etc/machine-id” mapping is required, because journald stores the logs in a directory named after the machine-id
- “-v /data/syslog-ng/conf/journal.conf:/etc/syslog-ng/syslog-ng.conf” maps the configuration from the host file system into the container. If you use a more complex configuration with includes or crypto keys, map a directory instead.
- “-v /data/syslog-ng/logs/:/var/log” maps the directory where we store logs on the host file system into the container
- “-v /var/log/journal:/var/log/journal” maps the directory with journal logs from the host file system into the container
- “–name journal” assigns the name “journal” to the container
- “balabit/syslog-ng:latest” is the name of the Docker image to start
When you first execute this command, it can take a few minutes until syslog-ng is up and running, because the image is downloaded over the Internet. On subsequent executions, Docker will use the local copy and start immediately.
Testing
You can check that syslog-ng is collecting logs properly by taking a look at the log files created by syslog-ng. There are two log files. If you used the above directory structure and configuration, /data/syslog-ng/logs/int contains the internal messages of syslog-ng. You should see similar logs in that file:
Jun 27 09:29:19 7e4fcd8247c4 syslog-ng[1]: syslog-ng starting up; version='3.10.1'
The file /data/syslog-ng/logs/journal.json has your log messages read from the journal in JSON format. It might be a huge file if you had many messages in the journal. The following is a sample log message, the same syslog-ng warning message together with CONTAINER_NAME and other docker related name-value pairs:
"{"journal":{"_UID":"0","_TRANSPORT":"journal","_SYSTEMD_UNIT":"docker.service","_SYSTEMD_SLICE":"system.slice","_SYSTEMD_CGROUP":"/system.slice/docker.service","_SOURCE_REALTIME_TIMESTAMP":"1498555672081993","_PID":"972","_MACHINE_ID":"c299b16e64ac46e6ac3c38ac5da988c0","_HOSTNAME":"localhost.localdomain","_GID":"0","_EXE":"/usr/bin/dockerd-current","_COMM":"dockerd-current","_CMDLINE":"/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --selinux-enabled --log-driver=journald --signature-verification=false","_CAP_EFFECTIVE":"1fffffffff","_BOOT_ID":"7b55ae658381491983949f5cf5274425","PRIORITY":"6","MESSAGE":"[2017-06-27T09:27:52.081729] WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode Please update it to use the syslog-ng 3.10 format at your time of convenience, compatibility mode can operate less efficiently in some cases. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file.;\r","CONTAINER_TAG":"","CONTAINER_NAME":"journal","CONTAINER_ID_FULL":"8c1a79c3d0a150102057cfff010ec72f4fd2cb383bc60a784018fe9c6b6ea721","CONTAINER_ID":"8c1a79c3d0a1"},"PROGRAM":"dockerd-current","PRIORITY":"info","PID":"972","MESSAGE":"[2017-06-27T09:27:52.081729] WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode Please update it to use the syslog-ng 3.10 format at your time of convenience, compatibility mode can operate less efficiently in some cases. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file.;\r","HOST":"58559f2f1026","FACILITY":"local0","DATE":"Jun 27 09:27:52"}"
What is next?
Once a log message is in syslog-ng you have practically endless possibilities. You can parse messages, enrich them or process them in many other ways. As a next step, you can make sure that only relevant messages are stored by using message filtering. Finally, you have to store the messages somewhere. It could be your central syslog-ng server, Elasticsearch, Kafka or one of the many other destinations supported by syslog-ng.
It is certainly not our goal to teach you Docker or syslog-ng in-depth in this blog post. If you are interested and would like to know more, check these resources:
- Docker documentation: https://docs.docker.com/
- syslog-ng documentation: https://www.syslog-ng.com/technical-documents
- source of syslog-ng Docker images: https://github.com/balabit/syslog-ng-docker
In the next blog of the Docker series, I talk about collecting logs from other containers that are not covered by Docker’s own log collecting method.
Read the entire series about logging in Docker using syslog-ng in a single white paper.