For most of its history, syslog-ng could only be used for collecting, processing and storing log messages. Not any more. The Redis and Riemann destinations are already a step into the direction of metrics-based monitoring, and the monitoring source combined with Graphite template support are the next.
Installing Graphite can be a huge task, if you want to install and configure each component manually. In this post I will demonstrate an easy way: using Docker. When I used Docker, Graphite was available within minutes and ready for use. For testing I used Fedora Rawhide, and to make my life easier, I disabled SELinux and iptables. Of course, neither of these are recommended in a production environment.
First, install and enable Docker:
# yum install docker-io # systemctl enable docker # systemctl start docker
Next, download and install Graphite inside Docker with the following command line:
# docker run -p 80:80 -p 2003:2003 -p 2004:2004 -p 7002:7002 -d nickstenning/graphite
By default, when starting a container, Docker does not expose the open network ports to the world, just to other Docker containers, using an internal network. Even when exporting, it does so by using random port numbers. The “-p” switch and the repeated port numbers mean that these ports are exported to the outside world using the original port numbers. Make sure that these ports are actually free. The name at the end of the command line refers to the name of the docker image. On Fedora this command is required only once, and Graphite is started automatically after the next reboot (I was told that this is not the case on all distributions).
You can read more about this Docker image at https://github.com/nickstenning/docker-graphite For testing purposes I did not configure permanent storage, or anything fancy, but you can try these as well if you are interested.
On the syslog-ng side, the latest syslog-ng and incubator packages will be required. For Fedora, these are available from my Copr repository at http://copr.fedoraproject.org/coprs/czanik/syslog-ng35/.
If you have not installed syslog-ng and the incubator packages on your machine yet, enable the repository and install the software. To do this, download http://copr.fedoraproject.org/coprs/czanik/syslog-ng35/repo/fedora-rawhide/czanik-syslog-ng35-fedora-rawhide.repo and place it into /etc/yum.repos.d/ Once the repository is enabled, install the packages by entering:
# yum install syslog-ng-incubator
This will also download syslog-ng as a dependency, if it has not yet been installed. There are also some other incubator packages, but this base package contains everything you will need to try Graphite. If you have not enabled syslog-ng yet, you can do it similarly to Docker.
Monitoring source in syslog-ng works by calling Lua scripts at predefined intervals. There are a couple of example scripts available in the sources and in the module author’s blog. The next Lua script is a slightly modified version of one of these:
values = { "wait_for_run", "sleep", "swapped", "free", "buffers",
"cache", "swapped_in", "swapped_out", "io_in", "io_out", "interrupts",
"context_switches", "user_time", "kernel_time", "idle",
"wait_for_io" }
-- source
function vmstat()
local result = {}
local f = assert(io.popen("vmstat -n 1 2 | tail -n 1", 'r'))
line = f:read("*all")
f:close()
local i = 1
local send = -1
local sstart = 0
while i < #values do
sstart, send = line:find("[^%s]+", send + 1)
result["vmstat."..values[i]] = line:sub(sstart, send)
i = i + 1
end
return result
end
Copy and paste it into your favorite text editor and save it to /etc/syslog-ng/vmstat.lua. Next, edit /etc/syslog-ng.conf and append the following few lines to it:
source s_monitor {
monitor(
monitor-freq(5)
monitor-func("vmstat")
monitor-script("/etc/syslog-ng/vmstat.lua")
);
};
destination d_graphite {
tcp( "172.16.177.139"
port(2003)
template("$(graphite-output --key vmstat.* )")
);
};
log {source(s_monitor); destination(d_graphite); };
This will send the output of the above vmstat.lua script to Graphite. You will have to change the IP address of your Graphite server. Once syslog-ng is reloaded, you should be able to see data pouring into Graphite. Open the IP/FQDN of Graphite in a web browser, click the “Graphite” folder icon on the left hand panel and you should see “vmstat” below it.
The following screenshot is from Graphite displaying performance statistics from a machine that is compiling software packages. Iit is easy to spot where compilation started and ended.
If you have any questions about the syslog-ng side, do not hesitate to contact us!