Actions

Tags

Peter Czanik
18 Mar 2021
Parsing Fortigate logs and other syslog-ng 3.31 news

Version 3.31 of syslog-ng has been released recently. One of its most user-visible features is the parser for Fortigate logs, yet another networking vendor that produces log messages not conforming to syslog specifications. Parsing Fortigate logs builds upon the new no-header flag of syslog-ng combined with the key-value and date parsers. Other features include a new silent message option for the Telegram destination and automatic directory creation for disk-buffer files.

Note: as you could guess from the previous paragraph, Fortigate is not alone. Cisco also has “interesting” log messages, and a bit of extra parsing also helps with PAN-OS, even if their messages conform to syslog specifications.

Before you begin

In this blog, we check out some of the new features of syslog-ng 3.31. If you want to test them, you need to install syslog-ng 3.31 or later (I’m running a Git snapshot version on my laptop). It is available already in FreeBSD ports, but most Linux distributions carry older versions. Check the 3rd-party binaries page on the syslog-ng website if there are up-to-date 3rd-party packages for your distribution of choice: https://www.syslog-ng.com/3rd-party-binaries

Once you have syslog-ng 3.31 up and running, you are ready to test some of the new features. Let’s start with some of the smaller features and finish with the big one!

Disk buffer directory

Prior to version 3.31 of syslog-ng, you had to create a directory for disk buffer files manually before starting syslog-ng. If the directory was missing, syslog-ng stopped with an error. In my latest syslog-ng article on opensoure.com, where syslog-ng on a Raspberry Pi machine buffers sensor data until the server becomes available, I also warn you to create the directory.

Starting with version 3.31, if the directory specified in the configuration does not exist, syslog-ng creates it automatically.

Silent Telegram messages

Sending alerts to Telegram from syslog-ng has been possible for years already. But there are some situations when you want to receive all the interesting messages on your mobile without being disturbed each time a new message arrives. This is where the new disable_notification() option to the Telegram destination can come handy. If set to `True`, your mobile receives the new messages without alerting you each time.

“no-header” flag for the syslog parser

Some log messages only keep the PRI field from the syslog specification, but they lack a proper syslog header. Among others, Fortigate logs belong to these log messages. In this case, you can use the “no-header” flag on incoming log messages. This way, syslog-ng parses the PRI field, and puts the rest of the log message into $MSG. If the message is otherwise structured, you can parse it further, extract the date using the date parser, and create name-value pairs from other useful information.

Working with Fortigate logs

Here are some sample Fortigate log messages. These were used to test the new parser, and we will use them as well. You can copy and paste them from here or from /usr/share/syslog-ng/include/scl/fortigate/fortigate.conf as well:

<189>date=2021-01-15 time=12:58:59 devname="FORTI_111" devid="FG100D3G12801312" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" eventtime=1610704739683510055 tz="+0300" srcip=91.234.154.139 srcname="91.234.154.139" srcport=45295 srcintf="wan1" srcintfrole="wan" dstip=213.59.243.9 dstname="213.59.243.9" dstport=46730 dstintf="unknown0" dstintfrole="undefined" sessionid=2364413215 proto=17 action="deny" policyid=0 policytype="local-in-policy" service="udp/46730" dstcountry="Russian Federation" srccountry="Russian Federation" trandisp="noop" app="udp/46730" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=5 craction=262144 crlevel="low"
<189>date=2021-01-15 time=12:58:59 devname="FORTI_111" devid="FG100D3G12801312" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" eventtime=1610704739683498829 tz="+0300" srcip=91.234.154.139 srcname="91.234.154.139" srcport=45295 srcintf="wan1" srcintfrole="wan" dstip=213.59.243.9 dstname="213.59.243.9" dstport=46730 dstintf="unknown0" dstintfrole="undefined" sessionid=2364413214 proto=17 action="deny" policyid=0 policytype="local-in-policy" service="udp/46730" dstcountry="Russian Federation" srccountry="Russian Federation" trandisp="noop" app="udp/46730" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=5 craction=262144 crlevel="low"
<189>date=2021-01-15 time=12:58:59 devname="FORTI_111" devid="FG100D3G12801312" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1610704739683525562 tz="+0300" srcip=10.9.1.26 srcname="sotina-sv" srcport=61105 srcintf="9 VLAN" srcintfrole="lan" dstip=77.88.55.66 dstname="www.yandex.ru" dstport=443 dstintf="wan1" dstintfrole="wan" sessionid=2364410752 proto=6 action="close" policyid=42 policytype="policy" poluuid="c1e5431a-d082-51e7-53e0-d3a8ab1a3ee2" service="HTTPS" dstcountry="Russian Federation" srccountry="Reserved" trandisp="snat" transip=213.59.243.9 transport=61105 appid=42899 app="Yandex" appcat="General.Interest" apprisk="elevated" applist="Application Control_User" duration=15 sentbyte=7286 rcvdbyte=1490 sentpkt=16 rcvdpkt=8 wanin=1158 wanout=6470 lanin=6470 lanout=6470 utmaction="allow" countapp=1 osname="Windows" srcswversion="7" unauthuser="sotina-sv" unauthusersource="kerberos" mastersrcmac="00:24:21:ac:fb:da" srcmac="00:24:21:ac:fb:da" srcserver=0
<189>date=2021-01-15 time=12:58:59 devname="FORTI_111" devid="FG100D3G12801312" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" eventtime=1610704739683532607 tz="+0300" srcip=94.143.50.155 srcname="94.143.50.155" srcport=56368 srcintf="wan1" srcintfrole="wan" dstip=213.59.243.9 dstname="213.59.243.9" dstport=46730 dstintf="unknown0" dstintfrole="undefined" sessionid=2364413216 proto=6 action="deny" policyid=0 policytype="local-in-policy" service="tcp/46730" dstcountry="Russian Federation" srccountry="Russian Federation" trandisp="noop" app="tcp/46730" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=5 craction=262144 crlevel="low"

As you can see, these lines start as any other log messages start, with a <PRI> field, but otherwise, they lack a proper syslog header. All further data is described as name-value pairs.

When we send this log to a port without Fortigate parsing enabled, the results are far from ideal. The date and time of the message will reflect the current time instead of the time encoded in the log message:

Mar 12 15:14:21 localhost date=2021-01-15 time=12:58:59 devname="FORTI_111" devid="FG100D3G12801312" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" eventtime=1610704739683510055 tz="+0300" srcip=91.234.154.139 srcname="91.234.154.139" srcport=45295 srcintf="wan1" srcintfrole="wan" dstip=213.59.243.9 dstname="213.59.243.9" dstport=46730 dstintf="unknown0" dstintfrole="undefined" sessionid=2364413215 proto=17 action="deny" policyid=0 policytype="local-in-policy" service="udp/46730" dstcountry="Russian Federation" srccountry="Russian Federation" trandisp="noop" app="udp/46730" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=5 craction=262144 crlevel="low"

By using the fortigate-parser(), we can extract the date from the message and also create name-value pairs from the message. This enables easier filtering (alerting), saving only specific fields to save space, or storing specific fields to an SQL or noSQL database, like Elasticsearch.

Luckily, there is no need to configure the fortigate-parser() yourself. Unless you have a high message rate, where even the smallest extra processing counts, using the default-network-drivers() of syslog-ng is easier. It parses Fortigate logs, and many other message types automatically. For this scenario and for other debugging ideas, check the Cisco parser blog at https://www.syslog-ng.com/community/b/blog/posts/parsing-cisco-logs-in-syslog-ng

The following configuration uses the default-network-drivers with a JSON-formatted destination file. This way, you can see that syslog-ng parses the logs and creates name-value pairs from them.

source s_net {
    default-network-drivers();
};
template t_jsonfile {
    template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs --key ISODATE)\n\n");
};
destination d_fromfortigate {
    file("/var/log/fromfortigate" template(t_jsonfile));
};
log {
    source(s_net);
    destination(d_fromfortigate);
};

The source is default-network-drivers() and it listens on TCP port 514. The template uses JSON formatting, includes any syslog fields and removes the leading dots from name-value pairs (by default syslog-ng parsers create names that start with a dot, but it can cause weird problems for example with Elasticsearch) before storing them. Then, apply this template to a destination file. Finally, a log statement connects the source to the destination.

Using netcat, you can send the test messages to the network source on port 514 and check the results:

czplaptop:~ # cat fortigate.txt | netcat -4 -n -N -v 127.0.0.1 514
Connection to 127.0.0.1 514 port [tcp/*] succeeded!
czplaptop:~ # cat /var/log/fromfortigate
{"fortigate":{"vd":"root","tz":"+0300","type":"traffic","trandisp":"noop","time":"12:58:59","subtype":"local","srcport":"45295","srcname":"91.234.154.139","srcip":"91.234.154.139","srcintfrole":"wan","srcintf":"wan1","srccountry":"Russian Federation","sessionid":"2364413215","service":"udp/46730","sentpkt":"0","sentbyte":"0","rcvdbyte":"0","proto":"17","policytype":"local-in-policy","policyid":"0","logid":"0001000014","level":"notice","eventtime":"1610704739683510055","duration":"0","dstport":"46730","dstname":"213.59.243.9","dstip":"213.59.243.9","dstintfrole":"undefined","dstintf":"unknown0","dstcountry":"Russian Federation","devname":"FORTI_111","devid":"FG100D3G12801312","date":"2021-01-15","crscore":"5","crlevel":"low","craction":"262144","appcat":"unscanned","app":"udp/46730","action":"deny"},"app":{"name":"fortigate"},"SOURCE":"s_net","PRIORITY":"notice","MESSAGE":"date=2021-01-15 time=12:58:59 devname=\"FORTI_111\" devid=\"FG100D3G12801312\" logid=\"0001000014\" type=\"traffic\" subtype=\"local\" level=\"notice\" vd=\"root\" eventtime=1610704739683510055 tz=\"+0300\" srcip=91.234.154.139 srcname=\"91.234.154.139\" srcport=45295 srcintf=\"wan1\" srcintfrole=\"wan\" dstip=213.59.243.9 dstname=\"213.59.243.9\" dstport=46730 dstintf=\"unknown0\" dstintfrole=\"undefined\" sessionid=2364413215 proto=17 action=\"deny\" policyid=0 policytype=\"local-in-policy\" service=\"udp/46730\" dstcountry=\"Russian Federation\" srccountry=\"Russian Federation\" trandisp=\"noop\" app=\"udp/46730\" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat=\"unscanned\" crscore=5 craction=262144 crlevel=\"low\"","ISODATE":"2021-01-15T12:58:59+01:00","HOST_FROM":"localhost","HOST":"localhost","FACILITY":"local7","DATE":"Jan 15 12:58:59"}

[…]

You can see all the name-value pairs extracted from the log messages, including a proper date as found in the log message.

What is next?

These were just my highlights from the 3.31 syslog-ng release. For the complete list of changes, check https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.31.1 where you might find other new features or bugfixes that might convince you to upgrade syslog-ng on your hosts.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @Pczanik.

Parents Comment Children
Related Content