7.4.1. Procedure – Prerequisites The syslog-ng Premium Edition 7 Administrator Guide
To send messages from syslog-ng PE to HDFS, complete the following steps.
If you want to use the Java-based modules of syslog-ng PE (for example, the Elasticsearch, HDFS, or Kafka destinations), download and install the Java Runtime Environment (JRE), 1.7 (or newer).
The Java-based modules of syslog-ng PE are tested and supported when using the Oracle implementation of Java. Other implementations are untested and unsupported, they may or may not work as expected.
Download the Hadoop Distributed File System (HDFS) libraries (version 2.x) from http://hadoop.apache.org/releases.html.
Extract the HDFS libraries into a temporary directory, then collect the various
.jarfiles into a single directory (for example,
/opt/hadoop/lib/) where syslog-ng PE can access them. You must specify this directory in the syslog-ng PE configuration file. The files are located in the various
libdirectories under the
share/directory of the Hadoop release package. (For example, in Hadoop 2.7, required files are
hdfs/lib/*, but this may change between Hadoop releases, so it is easier to copy every
.jarfile into a single directory.