syslog-ng documentation

Your main source of knowledge

The syslog-ng product family has an extensive documentation, covering everything from how to install a product to the most complex configuration and settings descriptions. If you cannot find an answer to your question, try the mailing list - our community is always eager to help.

syslog-ng Premium Edition

Contents

7.6. kafka: Publishing messages to Apache Kafka

Starting with version 5.4, syslog-ng PE can directly publish log messages to the Apache Kafka message bus, where subscribers can access them.

Note

In order to use this destination, syslog-ng Premium Edition must run in server mode. Typically, only the central syslog-ng Premium Edition server uses this destination. For details on the server mode, see Section 2.3.3, Server mode.

  • This destination is only supported on the Linux platforms that use the linux glibc2.11 installer, including: Debian 7 (wheezy), Red Hat ES 7, Ubuntu 12.04 (Precise Pangolin), Ubuntu 14.04 (Trusty Tahr).

  • Since syslog-ng PE uses the official Java Kafka producer, the kafka destination has significant memory usage.

  • The log messages of the underlying client libraries are available in the internal() source of syslog-ng PE.

Declaration: 

@module mod-java
@include "scl.conf"

kafka(
    client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:<path-to-preinstalled-kafka-libraries>")
    kafka-bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
    topic("${HOST}")

);
Example 7.16. Sending log data to Apache Kafka

The following example defines a kafka destination, using only the required parameters.

@module mod-java
@include "scl.conf"

destination d_kafka {
  kafka(
    client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/KafkaDestination.jar:/usr/share/kafka/lib/")
    kafka-bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
    topic("${HOST}")
  );
};