Sending logs from Logstash to syslog-ng

Logstash adds a new syslog header to log messages before forwarding them to a syslog server. In the case of syslog messages, it is problematic as there will be two syslog headers in the message. Using syslog-ng for everything logging related in an Elasticsearch environment can considerably simplify your architecture. Still, there are situations, when Filebeats and Logstash are already deployed and you need some logs from Logstash in syslog-ng. Learn how you can remove the extra syslog header.

Before you begin

This blog assumes that you utilize Filebeat to collect syslog messages, forward them to a central Logstash server, and Logstash forwards the messages to syslog-ng. If you collect other types of log messages, the syslog-ng configuration example does not apply to you. Of course you can use most of the configuration but only with slight modifications.

Configuring Filebeat

The following Filebeat configuration reads a single file – /var/log/messages – and sends its content to Logstash running on the same host:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/messages
output.logstash:
  hosts: ["localhost:5044"]

Configuring Logstash

Syslog output is available as a plugin to Logstash and it is not installed by default. Before you can utilize it, you have to install it. Go to your Logstash directory (/usr/share/logstash, if you installed Logstash from the RPM package), and execute the following command to install it:

bin/logstash-plugin install logstash-output-syslog

The following Logstash configuration collects messages from Beats and sends them to a syslog destination. You need to specify four parameters for the syslog destination and make sure that they are matched on the syslog-ng side:

• host: where to send logs
• port: which port to use
• protocol: tcp or udp
• rfc: which syslog message format to use

Read more about the possible parameters in the Logstash documentation at https://www.elastic.co/guide/en/logstash/current/plugins-outputs-syslog.html

input {
  beats {
    port => 5044
  }
}
output {
  syslog {
    host => "127.0.0.1"
    port => 514
    protocol => "tcp"
    rfc => "rfc3164"
  }
  stdout { codec => rubydebug }
}

The above configuration sends logs to port 514 using a TCP connection and the legacy message format.

The “stdout” destination facilitates debugging. When enabled, Logstash prints all log messages (nicely formatted) in the terminal where the application was started. You can remove the relevant line from the configuration once everything is working as expected.

Configuring syslog-ng

Last but not least, you also need to configure syslog-ng. You can append the configuration snippet below to the end of your current syslog-ng configuration or create a new .conf file under /etc/syslog-ng/conf.d/ if it is enabled in your distribution.

There are four major parts in this configuration snippet:

1. A network source with settings that match your Logstash settings. Port, protocol, message format – all can be changed but only if changed on both sides.
2. A file destination to store incoming messages.
3. A syslog parser to parse the message part again as a syslog message.
4. A log statement to connect all of the above parts together.

source s_logstash {
    tcp(port(514));
};
destination d_fromlogstash {
  file("/var/log/fromlogstash");
};
parser p_syslog {
  syslog-parser(flags(expect-hostname));
};
log {
  source(s_logstash);
  parser(p_syslog);
  destination(d_fromlogstash);
};

Note that if you collect something other than syslog messages using Filebeat, then you do not need the syslog parser. You can replace it with another parser, use another type of destination, whatever best fits the given log messages.

Testing

Compare the log file read by Filebeat with the log file written by syslog-ng. You should find all the messages from the source file in the destination file and the lines should be identical.

Note that if you use older syslog-ng versions (for example, version 3.5 from the EPEL repository), you might notice some minor differences. If a message does not contain a PID, like one coming from the kernel, syslog-ng might append a dash between square brackets [‒] to the program name.