Until now collecting logs behind proxies or load balancers needed some compromises. You either trusted the host information included in the log messages or you could only see the proxy as the sender host. Starting with syslog-ng 3.30 there is a third option available: using the PROXY protocol. While not an official Internet standard, it is supported by a number of popular software, like HAProxy. Other software can be extended to use it, like F5 load balancers using iRules. This way crucial information about the original network connection is not lost, but it is forwarded to the server by the proxy.
From this blog you can learn about the PROXY protocol, how to enable it in the syslog-ng configuration, and how to send test messages using loggen directly and through HAProxy.
Before you begin
You need to use at least sylog-ng version 3.30 (or syslog-ng PE 7.0.23 of the commercial version) to utilize PROXY protocol support. Most Linux distributions still carry older versions. You can find information about unofficial 3rd party syslog-ng repositories with up-to-date syslog-ng packages at https://www.syslog-ng.com/3rd-party-binaries. At the moment these versions are not yet released, so I used git snapshot packages for testing.
In my blog I will show you a simple configuration for HAProxy, as it is available for free and it is included in most Linux distributions. I ran my tests on three openSUSE virtual machines separately for the client sending logs, for HAProxy and for the syslog-ng server. But you can use any platform that HAProxy and syslog-ng supports and can actually have all three on a single host.
The PROXY protocol
Before we take a deep dive into syslog-ng configuration, let’s take a closer look at the PROXY protocol. The PROXY protocol was created by HAProxy developers and it is available on their website: http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt While it resembles an RFC, it is not an official standard, yet many devices and software support it because it solves a very common problem. When a TCP connection goes through a proxy or a load balancer, the original TCP information, including the source IP address, is lost. The PROXY protocol makes sure that this information reaches the servers behind the proxy. The PROXY protocol has two versions. The first version is text-based, while the second version forwards information in a binary form. The syslog-ng implementation supports the first version of the protocol.
When the text-based first version of the PROXY protocol is enabled, the proxy starts each new TCP connection with a similar line:
PROXY TCP4 192.168.0.1 192.168.0.11 56324 443
As you can see, it starts with a fixed text – PROXY – followed by the version of the TCP protocol, the source and destination IP addresses and ports of the connection. The PROXY protocol is not auto detected, it does not involve any heuristics. If the first line does not follow this format, the server side rejects the connection. The only exception is when the proxy sends the following line:
PROXY UNKNOWN
What happens here depends on the implementation: some servers simply reject these connections, but most accept them, including syslog-ng.
Configuring syslog-ng
Append the following configuration snippet to your syslog-ng.conf or place it in a file with .conf extension under the /etc/syslog-ng/conf.d/ directory, if syslog-ng on your host is configured to use it.
source s_tcp_pp {
network(
port(7777)
ip(0.0.0.0)
transport("proxied-tcp")
);
};
destination d_file {
file("/var/log/pp.log" template("$(format-json --scope nv-pairs)\n"));
};
log {
source(s_tcp_pp);
destination(d_file);
};
The source listens on port 7777 and expects incoming connections to use the PROXY protocol. You can also use encrypted connections here by replacing “proxied-tcp” with “proxied-tls” and adding the TLS-related options, just like with a regular encrypted source.
The file destination uses JSON formatting. This way you can see the name-value pairs created by this source: PROXIED_SRCPORT, PROXIED_SRCIP, PROXIED_IP_VERSION, PROXIED_DSTPORT and PROXIED_DSTIP. Note that these name-value pairs are not created with PROXY UNKNOWN.
Finally, the log statement connects the source and destination into a pipeline together. Reload syslog-ng for the new configuration to take effect.
Testing with loggen
The easiest way to test the above configuration is to use the loggen utility of syslog-ng. First try to send a few logs without enabling PROXY protocol support:
loggen -i -S localhost 7777
You will not find any new log messages in the new file destination. However, /var/log/messages will contain messages similar to these (if logging of the internal() source is enabled):
Nov 6 16:16:07 localhost syslog-ng[891]: PROXY proto header with invalid header length; max_parsable_length='216', max_length_by_spec='108', length='255', header='<38>2020-11-06T16:16:07 localhost prg00000[1234]: seq: 0000000000, thread: 0000, runid: 1604675767, stamp: 2020-11-06T16:16:07 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD\x0a<38>2020-11-06T16:16:07 localhost prg00000[1234]: seq: 0000000001, thread: 0000, runid: 1604675767, stamp: 2020-11-06T16:16:07 [...] PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD\x0a-client.c' Nov 6 16:16:07 localhost syslog-ng[891]: Error parsing PROXY protocol header; Nov 6 16:16:07 localhost syslog-ng[891]: Syslog connection closed; fd='16', client='AF_INET(127.0.0.1:41066)', local='AF_INET(0.0.0.0:7777)'
It means that loggen did not use the PROXY header and thus the connection was rejected. Let’s try again, this time using the new -H option of loggen:
loggen -i -S -H localhost 7777
This time /var/log/messages show a successful connection:
Nov 6 16:37:25 localhost syslog-ng[891]: Initializing PROXY protocol source driver; driver='0x560b2fb9b310' Nov 6 16:37:25 localhost syslog-ng[891]: Syslog connection accepted; fd='16', client='AF_INET(127.0.0.1:41068)', local='AF_INET(0.0.0.0:7777)' Nov 6 16:37:25 localhost syslog-ng[891]: PROXY protocol header parsed successfully; Nov 6 16:37:29 localhost syslog-ng[891]: Syslog connection closed; fd='16', client='AF_INET(127.0.0.1:41068)', local='AF_INET(0.0.0.0:7777)'
And in /var/log/pp.log you will find similar messages:
{"SOURCE":"s_tcp_pp","PROXIED_SRCPORT":"7075","PROXIED_SRCIP":"192.168.1.48","PROXIED_IP_VERSION":"4","PROXIED_DSTPORT":"514","PROXIED_DSTIP":"192.168.1.47","PROGRAM":"prg00000","PID":"1234","MESSAGE":"seq: 0000003961, thread: 0000, runid: 1604677045, stamp: 2020-11-06T16:37:29 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD","LEGACY_MSGHDR":"prg00000[1234]: ","HOST_FROM":"127.0.0.1","HOST":"localhost"}
{"SOURCE":"s_tcp_pp","PROXIED_SRCPORT":"7075","PROXIED_SRCIP":"192.168.1.48","PROXIED_IP_VERSION":"4","PROXIED_DSTPORT":"514","PROXIED_DSTIP":"192.168.1.47","PROGRAM":"prg00000","PID":"1234","MESSAGE":"seq: 0000003962, thread: 0000, runid: 1604677045, stamp: 2020-11-06T16:37:29 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD","LEGACY_MSGHDR":"prg00000[1234]: ","HOST_FROM":"127.0.0.1","HOST":"localhost"}
The PROXY protocol related name-value pairs contain random IP addresses and ports by default, but you can specify your own values as well if you want to test your configuration with specific values.
Installing and configuring HAProxy
HAProxy is part of most Linux distributions and it is also available on FreeBSD. On openSUSE the HAProxy package comes with a sample configuration. All it needs is appending four lines at the end. Append these lines:
listen sng bind *:6666 mode tcp server server1 172.16.167.153:7777 maxconn 32 send-proxy
Of course, you also need to change the IP address to the address of your syslog-ng server. The above configuration snippet listens on port 6666 and forwards connections to port 7777 on the given IP address. The “mode tcp” makes sure that HAProxy handles the connection as a generic TCP connection instead of as a HTTP connection. The “send-proxy” keyword enables PROXY protocol for this destination. If it is not included, syslog-ng rejects the connections coming from this proxy. Once you saved the new configuration and reloaded HAProxy, you are ready for testing!
Testing through HAProxy
You are not longer limited to using loggen when testing through HAProxy. You can use any software that can send logs to port 6666 through a TCP connection, but for testing, the generic logger and the loggen utility from syslog-ng are the easiest to use.
logger --tcp --port 6666 --server 172.16.167.139 --rfc3164 this is a test
or
loggen -i -S 172.16.167.139 6666
Of course, replace the IP address with the IP address of your HAProxy server. You should see logs in /var/log/pp.log with real IP addresses of your hosts:
{"SOURCE":"s_tcp_pp","PROXIED_SRCPORT":"59516","PROXIED_SRCIP":"172.16.167.1","PROXIED_IP_VERSION":"4","PROXIED_DSTPORT":"6666","PROXIED_DSTIP":"172.16.167.139","PROGRAM":"prg00000","PID":"1234","MESSAGE":"seq: 0000002318, thread: 0000, runid: 1604679827, stamp: 2020-11-06T17:23:48 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD","LEGACY_MSGHDR":"prg00000[1234]: ","HOST_FROM":"172.16.167.139","HOST":"localhost"}
{"SOURCE":"s_tcp_pp","PROXIED_SRCPORT":"59518","PROXIED_SRCIP":"172.16.167.1","PROXIED_IP_VERSION":"4","PROXIED_DSTPORT":"6666","PROXIED_DSTIP":"172.16.167.139","PROGRAM":"czanik","MESSAGE":"This is a test","LEGACY_MSGHDR":"czanik: ","HOST_FROM":"172.16.167.139","HOST":"czplaptop"}
One more thing
In your log analysis software, you most likely want to use the real source IP instead of the IP of the proxy server / load balancer. You can train your analytics software about the PROXIED_SRCIP name-value pair, but it is easier to handle this on the syslog-ng side, so I suggest rewriting HOST_FROM with the value of PROXIED_SRCIP. Here is a slightly modified version of the previous configuration with a rewrite rule added to it:
source s_tcp_pp {
network(
port(7777)
ip(0.0.0.0)
transport("proxied-tcp")
);
};
rewrite r_fixfrom {
set("$PROXIED_SRCIP", value("HOST_FROM"));
};
destination d_file {
file("/var/log/pp.log" template("$(format-json --scope nv-pairs)\n"));
};
log {
source(s_tcp_pp);
rewrite(r_fixfrom);
destination(d_file);
};
When you send another test message, HOST_FROM will now contain the real source IP address instead of the proxy IP address:
{"SOURCE":"s_tcp_pp","PROXIED_SRCPORT":"52532","PROXIED_SRCIP":"172.16.167.1","PROXIED_IP_VERSION":"4","PROXIED_DSTPORT":"6666","PROXIED_DSTIP":"172.16.167.139","PROGRAM":"czanik","MESSAGE":"This is a test fixed","LEGACY_MSGHDR":"czanik: ","HOST_FROM":"172.16.167.1","HOST":"czplaptop"}
What is next?
From this blog you could learn how to configure syslog-ng for the PROXY protocol and how to validate your configuration using loggen directly. I also showed you a very basic HAProxy configuration and an example for sending logs to syslog-ng through HAProxy. This setup was sufficient to test the PROXY protocol, but using a single server in a production environment does not make much sense.
If you need commercial level support to integrate syslog-ng with a proxy or load balancer like HAProxy or F5, consider buying syslog-ng PE which not only providesenterprise support, but also comes with a number of extra features. Do not hesitate to contact us at https://www.syslog-ng.com/products/log-management-software/
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.