Any time I see that one of the syslog-ng destinations has a major new version, I'm a bit scared, as it is not uncommon to introduce breaking changes with them. MongoDB 6.0, however, was a pleasant surprise. I gave it a quick try, and everything worked as expected. Along the way, I even learned about MongoDB Compass, an easy-to-use GUI for MongoDB databases.
Note that I only did a quick test of MongoDB 6.0. I did not try the new 6.0 specific features. All I did was sending logs from syslog-ng to MongoDB and browsing the collected logs using MongoDB Compass.
Before you begin
Version 6.0 of the MongoDB server is not yet available in most Linux distributions. You can download the MongoDB community server from the MongoDB website: https://www.mongodb.com/try/download/community
Various command line utilities are also available from the MongoDB website. However, instead of those, I installed MongoDB Compass, a GUI application to work with MongoDB databases. It is available at https://www.mongodb.com/try/download/compass
You also need syslog-ng 3.32 or later (earlier versions might also work, but I recommend 3.32 because it added template support and had many performance improvements) with MongoDB support enabled. Also note that SLES / openSUSE syslog-ng packages lack MongoDB, while on FreeBSD MongoDB support is missing from the pre-built packages. However, you can compile it yourself from ports. Fedora / RHEL and Ubuntu / Debian packages support MongoDB access from syslog-ng.
For my tests, I used Rocky Linux 8 in a virtual machine, as it is well supported both by MongoDB and syslog-ng. I used the currently available latest unofficial syslog-ng packages: https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng337/
Installation
I downloaded the MongoDB community server and Compass rpm packages from the MongoDB website and installed them using dnf. Using simple rpm works for the server, but Compass needs a few extra dependencies which are automagically installed when using dnf.
When installing syslog-ng, make sure that you also install MongoDB support. For Fedora / RHEL, the syslog-ng package only includes core functionality, so you must also install the syslog-ng-mongodb sub-package. For Ubuntu / Debian, it depends on how you install syslog-ng. If you installed the syslog-ng package, then MongoDB support is already there, as the syslog-ng package installs all sub-packages and their dependencies. However, if you installed the syslog-ng-core package, you also need to install the syslog-ng-mod-mongodb sub-package.
Configuration
I did not touch any of the MongoDB configurations. On the syslog-ng side, you need a few lines of configuration to send logs to MongoDB. The following configuration snippet forwards all locally generated logs to a locally installed MongoDB server. Place this with a .conf extension into the /etc/syslog-ng/conf.d/ directory, then reload syslog-ng so the configuration can take effect:
destination d_mongo {
mongodb(
uri("mongodb://localhost/syslog")
collection("messages")
workers(8)
);
};
log {source(s_sys); destination(d_mongo);};
This example assumes that you have a source called s_sys (the name of the default log source for local logs on Fedora / RHEL). Logs are sent to the MongoDB server on localhost to the database called “syslog” and the collection called “messages”. In the example, eight worker threads are configured, which can significantly speed up saving logs to MongoDB on a larger system.
Testing
As soon as you reload syslog-ng, logs should start flowing to MongoDB. You can use the logger utility to generate a few test messages, or loggen (the testing / benchmarking tool bundled with syslog-ng) to generate a larger amount of test messages.
You can use various command line tools to query / dump MongoDB databases. This time I installed MongoDB Compass, a nice graphical application for the desktop. With just a few mouse clicks, you can reach your logs with it.
What is next?
The above setup is good enough for initial testing. However, if you want to use the MongoDB destination of syslog-ng in production, you should also familiarize yourself with the various command line tools for MongoDB. You can learn more about how to scale a syslog-ng connection to MongoDB and use templates for collection names in one of my previous blogs: https://www.syslog-ng.com/community/b/blog/posts/mongodb-support-improved-in-syslog-ng-3-32
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.