7.1.2. How syslog-ng PE interacts with Elasticsearch
The syslog-ng PE application sends the log messages to the official Elasticsearch client library, which forwards the data to the Elasticsearch nodes. The way how syslog-ng PE interacts with Elasticsearch is described in the following steps.
After syslog-ng PE is started and the first message arrives to the
elasticsearchdestination tries to connect to the Elasticsearch server or cluster. If the connection fails, syslog-ng PE will repeatedly attempt to connect again after the period set in
If the connection is established, syslog-ng PE sends JSON-formatted messages to Elasticsearch.
flush-limitis set to 1: syslog-ng PE sends the message reliably: it sends a message to Elasticsearch, then waits for a reply from Elasticsearch. In case of failure, syslog-ng PE repeats sending the message, as set in the
retries()parameter. If sending the message fails for
retries()times, syslog-ng PE drops the message.
This method ensures reliable message transfer, but is slow (about 1000 messages/second).
flush-limitis higher than 1: syslog-ng PE sends messages in a batch, and receives the response asynchronously. In case of a problem, syslog-ng PE cannot resend the messages.
This method is relatively fast (depending on the size of
flush-limit, about 8000 messages/second), but the transfer is not reliable. In transport mode, over 5000-30000 messages can be lost before syslog-ng PE recognizes the error. In node mode, about 1000 messages can be lost.
concurrent-requestsis higher than 1, syslog-ng PE can send multiple batches simultaneously, increasing performance (and also the number of messages that can be lost in case of an error). For details, see Section concurrent-requests().