Version 3.29 of syslog-ng was released recently including a user-contributed feature: the panos-parser(). It is parsing log messages from PAN-OS (Palo Alto Networks Operating System). Unlike some other networking devices, the message headers of PAN-OS syslog messages are standards-compliant. However, if you want to act on your messages (filtering, alerting), you still need to parse the message part. The panos-parser() helps you create name-value pairs from the message part of the logs.
From this blog you can learn why it is useful to parse PAN-OS log messages and how to use the panos-parser().
Before you begin
In order to use the panos-parser(), you need to install syslog-ng 3.29 or later. Most Linux distributions feature earlier versions, but the https://www.syslog-ng.com/3rd-party-binaries page of the syslog-ng website has some pointers to 3rd party repositories featuring up-to-date binaries.
You also need some PAN-OS log messages. If you are reading this blog, you most likely have some Palo Alto Networks devices at hand and your ultimate goal is to collect logs from those devices. In this blog I will use the sample log messages found in the configuration snippet implementing the panos-parser(). Even if you have “real” PAN-OS logs, it is easier to get started configuring and testing syslog-ng this way.
Why is it useful?
As mentioned earlier, PAN-OS devices send completely valid syslog messages. You are not required to do any additional parsing on them. Syslog-ng can interpret the message headers without any additional configuration and save the logs properly:
Apr 14 16:48:54 localhost 1,2020/04/14 16:48:54,unknown,SYSTEM,auth,0,2020/04/14 16:48:54,,auth-fail,,0,0,general,medium,failed authentication for user \'admin\'. Reason: Invalid username/password. From: 10.0.10.55.,1718,0x0,0,0,0,0,,paloalto Apr 14 16:54:18 localhost 1,2020/04/14 16:54:18,unknown,CONFIG,0,0,2020/04/14 16:54:18,10.0.10.55,,set,admin,Web,Succeeded, deviceconfig system,127,0x0,0,0,0,0,,paloalto
If you look at these logs, you can see that the message part is a list of comma-separated values. That should be easy for the csv-parser(), but the two lists above have a different set of fields and there are a few more message types not included here. You can find the parsers describing these message types in /usr/share/syslog-ng/include/scl/paloalto/panos.conf or in the same directory under /usr/local on most Linux distributions. The panos-parser() can detect what type of log it is and create name-value pairs accordingly. If none of the types match, the parser drops the log. Once you have name-value pairs, it is much easier to create alerts (filters) in syslog-ng or reports in Kibana (if you use the Elasticsearch destination of syslog-ng).
Configuring syslog-ng
In most Linux distributions, syslog-ng is configured in a way so that you can extend it by dropping a configuration file with a .conf extension into the /etc/syslog-ng/conf.d/ directory. In other cases, simply append the below configuration to syslog-ng.conf.
source s_regular { tcp(port(5141)); };
source s_net {
default-network-drivers(flags(store-raw-message));
};
source s_panosonly { tcp(port(5140) flags(no-parse,store-raw-message)); };
template t_jsonfile {
template("$(format-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs --key ISODATE)\n\n");
};
parser p_panos { panos-parser(); };
destination d_frompanos {
file("/var/log/frompanos" template(t_jsonfile));
};
destination d_other {
file("/var/log/other");
};
destination d_raw {
file("/var/log/raw" template("${RAWMSG}\n"));
};
log {
source(s_regular);
destination(d_other);
};
log {
source(s_net);
destination(d_raw);
if ("${.app.name}" eq "panos") {
destination(d_frompanos);
} else {
destination(d_other);
};
};
log {
source(s_panosonly);
destination(d_raw);
parser(p_panos);
destination(d_frompanos);
};
If you follow my blogs, the above configuration will look familiar: it is based on the configuration I prepared for Cisco in one of my recent blogs, I just replaced the Cisco-specific parts. Some of the text below is also reused, but there are some obvious differences, as the purpose of the parsers is different.
In the following section we will go over this configuration in detail, in the order the statements appear in the configuration. To make your life easier, I copied the relevant snippets below before explaining them.
Sources
source s_regular { tcp(port(5141)); };
This is a pretty regular TCP legacy syslog source on a random high port that I used to create the logs in the “Why is it useful” section. By default, the tcp() source handles all incoming log messages as if they were legacy (RFC3164) formatted and in case of PAN-OS logs it results in properly formatted logs.
source s_net {
default-network-drivers(flags(store-raw-message));
};
The default-network-drivers() source driver is a kind of a wild card. It opens different UDP and TCP ports using both the legacy and the new RFC5424 syslog protocols. Instead of just expecting everything to be regular syslog, it attempts a number of different parsers on incoming logs, including the panos-parser(). Of course, the extra parsing creates some overhead, but it is not a problem unless you have a very high message rate.
The store-raw-message flag means that syslog-ng preserves the original log message as is. It might be useful for debugging or if a log analysis software expects unmodified PAN-OS messages.
source s_panosonly { tcp(port(5140) flags(no-parse,store-raw-message)); };
The third source, as its name also implies, is only for PAN-OS log messages. Use it when you have a high message rate, you only send PAN-OS log messages at the given port and you are sure that the panos-parser() of syslog-ng can process all of your PAN-OS logs correctly. The no-parse flag means that incoming messages are not parsed automatically as they arrive. As you will see later, panos-parser() parses the incoming messages.
Templates
template t_jsonfile {
template("$(format-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs --key ISODATE)\n\n");
};
This is the template, which is often used together with Elasticsearch (without the line breaks at the end). This blog does not cover Elasticsearch, as there are many other blogs covering the topic. However, this template using the JSON template function is also useful here because it shows the name-value pairs parsed from PAN-OS log messages. These JSON formatted logs include all syslog-related fields, name-value pairs parsed from the message, and the date using the ISO standardized format. There is one little trick that might confuse you: the rekey and shift part removes the dots from the front of the name-value pair names. Syslog-ng uses the dot for name-value pairs created by parsers in the syslog-ng configuration library (SCL).
Parsers
parser p_panos { panos-parser(); };
This parser can extract name-value pairs from the log messages of PAN-OS devices. By default, these are stored in name-value pairs starting with .panos., but you can change the prefix using the prefix() parameter. Note that the panos-parser() drops the message if it does not match the rules. This can be a problem when you use it directly instead of using the default-network-drivers() where messages go through a long list of parsers.
Destinations
destination d_frompanos {
file("/var/log/frompanos" template(t_jsonfile));
};
This is a file destination where logs are JSON-formatted using the template from above. This way you can see all the name-value pairs syslog-ng creates. We use it for PAN-OS log messages.
destination d_other {
file("/var/log/other");
};
This is a flat file destination using regular syslog formatting. We use it to store non-PAN-OS log messages.
destination d_raw {
file("/var/log/raw" template("${RAWMSG}\n"));
};
This is yet another file destination. The difference here is that it uses a special template defined in-line. Using the RAWMSG macro, syslog-ng stores the log message without any modifications. This possibility is enabled by utilizing the store-raw-message flag on the source side and it is useful for debugging or when a SIEM or any other analysis software needs the original log message.
Log statements
Previously we defined the building blocks of the configuration. Using the log statements below,we are going to connect these building blocks together. The different building blocks can be used multiple times in the same configuration.
log {
source(s_regular);
destination(d_other);
};
This is the simplest log statement: it connects the regular tcp() source with a flat file destination. You can see the results from this in the “Why is it useful” section. The logs look OK, but at a closer look you can also see that they contain tons of structured information. That is where the panos-parser() comes in handy.
log {
source(s_net);
destination(d_raw);
if ("${.app.name}" eq "panos") {
destination(d_frompanos);
} else {
destination(d_other);
};
};
Unless you have a high message rate, above 50,000 to 200,000 events per second (EPS), or not enough hardware capacity, the recommended way of receiving PAN-OS messages is using the default-network-drivers(). It uses slightly more resources, but if a message does not match the expectations of the panos-parser(), it is still kept. The above log statement receives the message and then stores it immediately in raw format. It can be used for debugging and disabled later.
The if statement below checks if the message is recognized as a PAN-OS log message. If it is, the log is saved as a JSON formatted file. If not, it is saved as a flat file.
Note that the name-value pair is originally called .app.name, but in the output it appears as app.name, as the template removes the dot in front.
log {
source(s_panosonly);
destination(d_raw);
parser(p_panos);
destination(d_frompanos);
};
If you have a high message rate and you are sure that the panos-parser() detects all of your logs, you can use this solution to collect logs from your PAN-OS devices. For safety and debugging purposes I inserted the raw file destination in front of the parser. This way you can compare the number of lines in the two different file destinations. If they are not equal, check the logs that the panos-parser() discarded.
Testing
For testing I used a few sample log messages from the syslog-ng configuration snippet containing the panos-parser() and saved these messages into a file in /root/panoslogs.txt. Logger generated the regular syslog messages and netcat submitted the PAN-OS messages. Here is the content of /root/panoslogs.txt:
<12>Apr 14 16:48:54 paloalto.test.net 1,2020/04/14 16:48:54,unknown,SYSTEM,auth,0,2020/04/14 16:48:54,,auth-fail,,0,0,general,medium,failed authentication for user \'admin\'. Reason: Invalid username/password. From: 10.0.10.55.,1718,0x0,0,0,0,0,,paloalto <14>Apr 14 16:54:18 paloalto.test.net 1,2020/04/14 16:54:18,unknown,CONFIG,0,0,2020/04/14 16:54:18,10.0.10.55,,set,admin,Web,Succeeded, deviceconfig system,127,0x0,0,0,0,0,,paloalto
You will get different results epending on to which port you send the logs. You have already seen what happens when you send logs to port 5141. It looks great, but you can also see that a bit of parsing could do wonders to it.
Here is an example of sending logs to port 514:
logger -T --rfc3164 -n 127.0.0.1 -P 514 this is a regular syslog message cat /root/panoslogs.txt | netcat -4 -n -N -v 127.0.0.1 514
In this case you should see in the files:
localhost:/etc/syslog-ng/conf.d # cat /var/log/other
Sep 14 15:25:54 localhost root: this is a regular syslog message
localhost:/etc/syslog-ng/conf.d # cat /var/log/frompanos
{"panos":{"vsys_name":"","vsys":"","type":"SYSTEM","time_generated":"2020/04/14 16:48:54","subtype":"auth","severity":"medium","serial":"unknown","seqno":"1718","receive_time":"2020/04/14 16:48:54","opaque":"failed authentication for user \\'admin\\'. Reason: Invalid username/password. From: 10.0.10.55.","object":"","module":"general","future_use4":"0","future_use3":"0","future_use2":"0","future_use1":"1","eventid":"auth-fail","dg_hier_level_4":"0","dg_hier_level_3":"0","dg_hier_level_2":"0","dg_hier_level_1":"0","device_name":"paloalto","actionflags":"0x0"},"app":{"name":"panos"},"SOURCE":"s_net","RAWMSG":"<12>Apr 14 16:48:54 paloalto.test.net 1,2020/04/14 16:48:54,unknown,SYSTEM,auth,0,2020/04/14 16:48:54,,auth-fail,,0,0,general,medium,failed authentication for user \\'admin\\'. Reason: Invalid username/password. From: 10.0.10.55.,1718,0x0,0,0,0,0,,paloalto","PROGRAM":"paloalto_panos","PRIORITY":"warning","MESSAGE":"1,2020/04/14 16:48:54,unknown,SYSTEM,auth,0,2020/04/14 16:48:54,,auth-fail,,0,0,general,medium,failed authentication for user \\'admin\\'. Reason: Invalid username/password. From: 10.0.10.55.,1718,0x0,0,0,0,0,,paloalto","ISODATE":"2020-04-14T16:48:54+02:00","HOST_FROM":"localhost","HOST":"paloalto.test.net","FACILITY":"user","DATE":"Apr 14 16:48:54"}
{"panos":{"vsys_name":"","vsys":"","type":"CONFIG","time_generated":"2020/04/14 16:54:18","subtype":"0","serial":"unknown","seqno":"127","result":"Succeeded","receive_time":"2020/04/14 16:54:18","path":" deviceconfig system","host":"10.0.10.55","future_use2":"0","future_use1":"1","dg_hier_level_4":"0","dg_hier_level_3":"0","dg_hier_level_2":"0","dg_hier_level_1":"0","device_name":"paloalto","cmd":"set","client":"Web","admin":"admin","actionflags":"0x0"},"app":{"name":"panos"},"SOURCE":"s_net","RAWMSG":"<14>Apr 14 16:54:18 paloalto.test.net 1,2020/04/14 16:54:18,unknown,CONFIG,0,0,2020/04/14 16:54:18,10.0.10.55,,set,admin,Web,Succeeded, deviceconfig system,127,0x0,0,0,0,0,,paloalto","PROGRAM":"paloalto_panos","PRIORITY":"info","MESSAGE":"1,2020/04/14 16:54:18,unknown,CONFIG,0,0,2020/04/14 16:54:18,10.0.10.55,,set,admin,Web,Succeeded, deviceconfig system,127,0x0,0,0,0,0,,paloalto","ISODATE":"2020-04-14T16:54:18+02:00","HOST_FROM":"localhost","HOST":"paloalto.test.net","FACILITY":"user","DATE":"Apr 14 16:54:18"}
localhost:/etc/syslog-ng/conf.d # cat /var/log/raw
<13>Sep 14 15:25:54 localhost root: this is a regular syslog message
<12>Apr 14 16:48:54 paloalto.test.net 1,2020/04/14 16:48:54,unknown,SYSTEM,auth,0,2020/04/14 16:48:54,,auth-fail,,0,0,general,medium,failed authentication for user \'admin\'. Reason: Invalid username/password. From: 10.0.10.55.,1718,0x0,0,0,0,0,,paloalto
<14>Apr 14 16:54:18 paloalto.test.net 1,2020/04/14 16:54:18,unknown,CONFIG,0,0,2020/04/14 16:54:18,10.0.10.55,,set,admin,Web,Succeeded, deviceconfig system,127,0x0,0,0,0,0,,paloalto
When you send logs to port 5140 where the panos-parser() is the only parser, the results should be pretty similar. The only difference is that the regular syslog message is only saved to the raw file used for debugging, as it is discarded by the panos-parser().
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For contact information, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.