Recently, many services provide an HTTP-based API to send messages. With a bit of luck, the given service is already supported directly by syslog-ng, or by using the Apprise Python library from the syslog-ng Python destination. In other cases, you need to do the research yourself on how the given HTTP-based service works. It might be scary at first, but often, it just takes a bit of experimenting and reading the documentation.
In this blog, I’m showing you how to send log messages to Seq, a container-based log management software for application logs. The focus of this blog is to understand what to look for in the documentation of software to create an http()-based destination in syslog-ng. You can install Seq in a container, it is easy, but not necessary.
Before you begin
As I mentioned, installing Seq is optional if you also want to test the new syslog-ng configuration. It is easy, as long as you have Docker available on your host: https://docs.datalust.co/docs/getting-started-with-docker
On the syslog-ng side, any recent syslog-ng version should be OK. In this case, recent means syslog-ng version 3.21 or later. Check our 3rd-party packages page for up-to-date packages for your distro.
Reading the docs
The first step is getting familiar with the syslog-ng http() destination. The documentation is available at https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.30/administration-guide/38#TOPIC-1594993 Yes, it includes many pages of information. You do not have to read all possible options in depth. It is enough to skim through them and try to understand the various example configurations included next to the descriptions. Some of these are http()-specific, while others should already look familiar, like the disk-buffer or the macros in various template-related options.
By the end, you will notice that all examples are slightly different but most of them include at least the following lines:
destination d_http {
http(
url("http://127.0.0.1:8000")
method("PUT")
user-agent("syslog-ng User Agent")
body("${ISODATE} ${MESSAGE}")
);
};
The next step is reading the Seq documentation. Information is often scattered in small parts across multiple pages in the documentation. Luckily, the Seq documentation has a single page collecting all the necessary information: https://docs.datalust.co/docs/posting-raw-events
It is not an easy read at first, but it is not too long. Right at the top, you can already see two important parameters to be included in the http() destination:
-
method: POST
-
url: https://localhost:5341/api/events/raw?clef
Reading a bit further, we can find another important piece of information:
-
a required header: "Content-Type: application/vnd.serilog.clef"
Now comes the difficult part. The example shown on the page is not something you can easily produce in syslog-ng. By default, Seq expects a message template and then a list of name-value pairs to fill in the blanks in the template. You could create such log messages using the PatternDB parser of syslog-ng, but even then, it could be a painfully slow process.
Luckily, reading the documentation further leads us to a list of mandatory and optional fields used in the JSON-formatted message. While all the examples show the use of message templates, there is another option, using @m for a fully rendered message describing the event. That is something we have in syslog-ng, alongside the values optionally parsed from log messages.
There is one more field we must use: @t for the time (and date) of the message in ISO format. Anything else we send is optional from the Seq point of view.
Configuring the http() destination
Now that we have all information at hand, we can start configuring syslog-ng. Here is the configuration I came up with for my environment:
destination d_http {
http(
url("http://172.16.167.139:5341/api/events/raw?clef")
method("POST")
user-agent("syslog-ng User Agent")
headers("Content-Type: application/vnd.serilog.clef")
body('$(format-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs
--exclude DATE @t=${ISODATE} @m=${MESSAGE})')
);
};
The URL should look familiar, I only replaced the IP address with an actual server IP in my test environment. Next, we configure the method. Configuring the user-agent is not strictly necessary, but as it is in the examples and some services expect it to be filled, I use it all the time. Next, headers are configured.
The message body comes last. As expected, we use the format-json template function to create JSON formatted log messages. We include all the regular syslog fields (rfc5414) and all name-value pairs parsed from the log message. We remove the leading dot from name-value pair names that start with a dot. The date macro is excluded, as instead of that, we include the date with the proper formatting as @t. The original message is included in @m.
Testing
The focus of this blog was to help you to better understand what to look for in documentation. But, of course, you might also want to try the configuration we put together.
First of all, you need to include the freshly configured http() destination in a log statement and connect it with a source. For example, in case of openSUSE, the name of the local log source in the default syslog-ng configuration is called src:
log {
source(src);
destination(d_http);
};
Recent versions of syslog-ng parse sudo log messages by default. So, without any further configuration you can also send some parsed log messages to Seq, just by issuing a few commands through sudo. You can see an example screenshot below:
What is next?
Now that you followed how to create an http()-based destination for Seq, it is time to work on your own http()-based destination. As you could see, creating a new destination takes some reading and experimenting. If you work on something new and you get stuck, you can reach out to the syslog-ng community for help. And if you succeed creating a new destination you are welcome to share it with the community to make other users’ lives easier!
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @Pczanik.