Using a proxy with the http() destination of syslog-ng

The http() destination is quickly becoming one of the most often used destinations within syslog-ng. You might already be using it even if you are not aware of it. Quite a few syslog-ng destination drivers are actually just configuration snippets in the syslog-ng configuration library (SCL), utilizing the http() destination in the background. Just think about elasticsearch-http(), different Logging as a Service (LasS) providers, or slack(). Starting with syslog-ng version 3.28.1 you can also reach these services when there is a proxy server between syslog-ng and your destination.

In this blog, I show you a simple configuration that sends logs from syslog-ng through a Squid proxy server to an Elasticsearch destination.

Before you begin

First of all, you need at least syslog-ng version 3.28.1 with http() support enabled. As it was released last week, there is a good chance that it is not yet available as an official package for your distribution. Currently I am only aware of FreeBSD ports and Gentoo Linux. For others, check our 3^rd party binaries page, where you can find pointers to repositories with up-to-date syslog-ng versions. The FreeBSD package supports http() by default, but depending on which Linux distribution you install syslog-ng on, http() support is most likely available in a sub-package. It is called syslog-ng-http on Fedora/RHEL, syslog-ng-curl on openSUSE/SLES, and syslog-ng-mod-http on Debian/Ubuntu.

I used Squid as a proxy server and Elasticsearch as a destination. You can use anything else you like for proxy and HTTP server, but these are probably the easiest to get started with for a quick test.

Configuration

First of all, you need to setup an HTTP destination and a proxy server. I installed Elasticsearch 7.8 on CentOS 8 and Squid 4.11 on openSUSE Leap 15.1, but neither the host OS, nor the version really matter from the testing point of view. The installation and configuration of these software is outside of the scope of this blog, but should be straightforward even with basic admin experience. Take a note of the hostname/IP address and port number both for the proxy server and for the Elasticsearch server.

The only place where the version matters, is syslog-ng. You should install syslog-ng 3.28.1, or later. I installed it on CentOS 7 from my unofficial syslog-ng repo.

You can find the configuration I used to test the new proxy() feature below. Replace the IP address and port with the values you noted down earlier. Note that s_sys is the default local log source in my Fedora/CentOS packages, but it might be called something completely different on your system.

destination d_elasticsearch_http {
    elasticsearch-http(
        index("syslog-ng")
        type("")
        url("http://192.168.1.128:9200/_bulk")
        proxy("http://172.16.167.141:3128/")
    );
};

log {
    source(s_sys);
    destination(d_elasticsearch_http);
};

Append the configuration to syslog-ng.conf or create a new configuration with a .conf extension under /etc/syslog-ng/conf.d/ if your distribution supports it.

Testing

Reload syslog-ng for the configuration to take effect. Now check Elasticsearch to see if logs arrived. I used Kibana for that, but using the API directly should also work fine. Next, check if logs really went through the proxy server. Take a look at your Squid logs. You should see similar lines:

linux-yv1e:~ # tail -2 /var/log/squid/access.log
1593171069.796     80 172.16.167.137 TCP_MISS/200 898 POST http://192.168.1.128:9200/_bulk - HIER_DIRECT/192.168.1.128 application/json
1593171069.796     81 172.16.167.137 TCP_MISS/200 898 POST http://192.168.1.128:9200/_bulk - HIER_DIRECT/192.168.1.128 application/json

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.