MongoDB support improved in syslog-ng 3.32

MongoDB is one of the most popular NoSQL databases. Support for MongoDB was added to syslog-ng almost a decade ago. It was the first syslog-ng destination where you could store arbitrary name-value pairs. The performance of MongoDB has improved considerably over the years, but syslog-ng was not keeping up. Version 3.32 of syslog-ng is a huge step in the right direction. It is not only performance that was improved, but flexibility as well: you can now use templates in collection names.

Before you begin

You must use at least syslog-ng version 3.32 to access the MongoDB destination related improvements. Check our 3^rd party binaries page if your platform of choice does not have this syslog-ng version included yet.

The MongoDB destination driver uses the mongo-c-driver client library to access MongoDB. There were some incompatible changes in it. If you compile syslog-ng with MongoDB support yourself, make sure that you use at least version 1.11 of mongo-c-driver or patch syslog-ng sources with this fix. The FreeBSD ports has an older mongo-c-driver version. So, sysutils/syslog-ng will feature the above-mentioned fix for the syslog-ng 3.32 port, once available.

Improving performance

Previous versions of the MongoDB destination driver were single threaded. Even on many-core servers, only a single CPU core was utilized to send log messages. Starting with syslog-ng 3.32, you can define the number of worker threads in the configuration. In my experience, the performance of the MongoDB destination increased almost linearly together with the number of available CPU cores. You can set the number of threads using the workers() option:

destination d_mongo {
  mongodb(
    uri("mongodb://localhost/syslog")
    collection("messages")
    workers(8)
  );
};

Improving flexibility

In previous versions of the MongoDB destination driver, the name of the collection was a fixed value. Starting with syslog-ng 3.32 you can use a template to define the collection name. For example, you can collect messages from different hosts into different collections:

destination d_mongo {
  mongodb(
    uri("mongodb://localhost/syslog")
    collection("${HOST}_messages")
  );
};

Testing

Using the following configuration, you can test both the performance changes and templates in collection names. Just make sure that MongoDB is up and running and your test host has at least two CPU cores. Append it to syslog-ng.conf or put it in a .conf file under /etc/syslog-ng/conf.d/ if syslog-ng in your Linux distribution of choice supports it.

source s_net {tcp(port(514));};

destination d_mongo {
  mongodb(
    uri("mongodb://localhost/syslog")
    collection("${HOST}_messages")
    workers(8)
  );
};
log {source(s_net); destination(d_mongo); flags(flow-control);};

For the first tests, change workers() to 1, then increase it to the number of the CPU cores in your test host. Use loggen (part of the syslog-ng package) for testing:

loggen -i -S -r 10000000 localhost 514

You can also follow the MongoDB side using the mongostat utility. The message rates shown by loggen and mongostat are similar. The mongo utility can print your collection names:

[root@localhost ~]# mongo
MongoDB shell version v4.4.6
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("3ec96bec-5a3f-4c03-b3d4-fd9a9f361478") }
MongoDB server version: 4.4.6
---
The server generated these startup warnings when booting:
        2021-05-17T11:25:17.524+02:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
[…]
---
> use syslog
switched to db syslog
> show collections
localhost_messages
messages
>

Here you could see both “messages” and “localhost_messages”, as I added the HOST macro to the name of the collection.

-

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @Pczanik.