7.3.2. How syslog-ng OSE interacts with Elasticsearch
The syslog-ng OSE application sends the log messages to the official Elasticsearch client library, which forwards the data to the Elasticsearch nodes. The way syslog-ng OSE interacts with Elasticsearch is described in the following steps.
After syslog-ng OSE is started and the first message arrives to the
elasticsearch2destination tries to connect to the Elasticsearch server or cluster. If the connection fails, syslog-ng OSE will repeatedly attempt to connect again after the period set in
If the connection is established, syslog-ng OSE sends JSON-formatted messages to Elasticsearch.
flush-limitis set to 1: syslog-ng OSE sends the message reliably: it sends a message to Elasticsearch, then waits for a reply from Elasticsearch. In case of failure, syslog-ng OSE repeats sending the message, as set in the
retries()parameter. If sending the message fails for
retries()times, syslog-ng OSE drops the message.
This method ensures reliable message transfer, but is slow (about 1000 messages/second).
flush-limitis higher than 1: syslog-ng OSE sends messages in a batch, and receives the response asynchronously. In case of a problem, syslog-ng OSE cannot resend the messages.
This method is relatively fast (depending on the size of
flush-limit, about 8000 messages/second), but the transfer is not reliable. In transport mode, over 5000-30000 messages can be lost before syslog-ng OSE recognizes the error. In node mode, about 1000 messages can be lost.
concurrent-requestsis higher than 1, syslog-ng OSE can send multiple batches simultaneously, increasing performance (and also the number of messages that can be lost in case of an error). For details, see Section concurrent-requests().
Version 3.10 and newer of syslog-ng OSE automatically converts the timestamp (date) of the message to UTC, as needed by Elasticsearch and Kibana.