Actions

Tags

Peter Czanik
28 Nov 2018
Using syslog-ng to forward SNMP traps to Splunk

As we learned at the Splunk .conf18 this October, forwarding SNMP traps to Splunk can be a challenging task. Luckily, using syslog-ng can simplify it for us. All we need to make sure about is that snmptrapd logs traps to a file. The syslog-ng application can read and parse that file and forward the traps to Splunk HEC (HTTP Event Collector).

Before you begin

  • First, you need snmptrapd, the component that actually collects the SNMP traps.

  • You also need an up-to-date syslog-ng release running on the same machine. The snmptrap() source first appeared in version 3.16 of syslog-ng, so this version is the minimum requirement for this process. To access the latest enhancements to the http() destination, you have to upgrade to version 3.18.

  • Finally, you also need version 7 of Splunk in order to use HEC.

Configuring snmptrapd

The first step is configuring snmptrapd to log traps to a file in the format expected by syslog-ng. For more information about formatting the output depending on the SNMP version you are using, check the documentation at https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/release-notes/snmptrap-read-net-snmp-traps.

In the example below, I used SNMP version 2, so my /etc/snmp/snmptrapd.conf contains:

authCommunity log,execute,net public
format2 %.4y-%.2m-%.2l %.2h:%.2j:%.2k %B [%b]:\n%v\n
outputOption s

You can set the file name (where to save the logs) only on the command line. How you have to configure it depends on your Linux distribution. On CentOS 7 I appended the following line to /etc/sysconfig/snmptrapd:

OPTIONS="-Lf /var/log/snmptrapd.log"

Once you restart snmptrapd and collect a few traps, you should see them in the /var/log/snmptrapd.log file defined above.

Configuring Splunk

First I tried to configure syslog-ng to forward SNMP traps to Splunk HEC using traditional syslog formatting. It worked, but the fields parsed automatically by Splunk from the message were, frankly, all over the place. Instead, for better results, I strongly recommend parsing SNMP traps using the snmptrap() source of syslog-ng and using JSON formatting to forward the generated name-value pairs to Splunk.

Splunk HEC needs a token in log messages to gather information about their format and intended destination. You can create a token through the HEC page in Splunk by clicking the New Token button in the upper right corner.

After naming your token, select the “_JSON” source type on the next screen. At the end of the process, you will see a similar token: 1d56aa3f-b203-4254-8db7-25c395973bf3. Make sure you keep this tab open and make a note of the token information just to be on the safe side, as you will need this tab later on, especially in the syslog-ng configuration.

Configuring syslog-ng

Before sending logs to Splunk (or any other network destination, for that matter), I normally run a quick test during which I save the results to a local file. This way I can make sure that the rest of the configuration works fine and later I only need to focus on the final step. This makes troubleshooting easier.

In the configuration example below I combine the file and Splunk destinations in a single configuration. You can remove or comment out the file destination once you do not need it any more. Append this to your syslog-ng.conf file or save it to a separate file under the /etc/syslog-ng/conf.d/ directory (if your configuration allows doing so).

destination d_splunk {
    http(url("http://localhost:8088/services/collector/raw")
        method("POST")
        user_agent("syslog-ng")
        user("user")
        password("1d56aa3f-b203-4254-8db7-25c395973bf3")
        body("$(format-json --scope all-nv-pairs)")
    );
};

log {
  source {snmptrap(filename("/var/log/snmptrapd.log"));};
  destination {file("/var/log/example.log");};
  destination(d_splunk);
};

Note: the example above uses my data. Make sure you replace the value of “password” with the token in your Splunk configuration (check the tab you left open earlier).

If you take a closer look at the log statement, you will see something unusual. Instead of declaring the snmptrap source and the file destination separately, they are now declared inline in the log statement. This makes the configuration a lot more compact and easier to read. Naturally, you can only configure syslog-ng this way if the building blocks are used only once during configuration.

The only mandatory parameter in the snmptrap() source is the filename where the SNMP traps are collected by snmptrapd. By default, the name part of the name-value pairs receives the “.snmp.” prefix. You can change that default setting by using the prefix() option.

The d_splunk destination sends logs to Splunk HEC using JSON formatting. The “--scope all-nv-pairs” option forwards all available name-value pairs to Splunk. For more information on how to limit the scope to a smaller set of name-value pairs, check the documentation at https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/9#TOPIC-956417.

Version 3.18 of syslog-ng has many HTTP destination-related performance improvements. You can read more about these improvements in the documentation at https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.18/administration-guide/36#TOPIC-1044019

Testing

To test your configuration, send snmptrapd a message:

snmptrap -v2c -c public 127.0.0.1 666 NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification netSnmpExampleHeartbeatRate i 60 netSnmpExampleString s "string"

You can check if the message arrived by looking at the file destination. You should see the line below:

[root@centos7 ~]# tail -1 /var/log/example.log 
Nov 27 12:51:58 centos7 snmptrapd: hostname='localhost', transport_info='UDP: [127.0.0.1]:47357->[127.0.0.1]:162', sysUpTimeInstance='(666) 0:00:06.66', snmpTrapOID.0='netSnmpExampleHeartbeatNotification', netSnmpExampleHeartbeatRate='60', netSnmpExampleString='string'

Next, check the Splunk web interface. You should see a similar message:

splunk snmp json

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/balabit/syslog-ng. On Twitter, I am available as @PCzanik.

Related Content