syslog-ng documentation

Your main source of knowledge

The syslog-ng product family has an extensive documentation, covering everything from how to install a product to the most complex configuration and settings descriptions. If you cannot find an answer to your question, try the mailing list - our community is always eager to help.

syslog-ng Premium Edition

Contents

11.1.7. Template functions of syslog-ng PE

The following template functions are available in syslog-ng PE.

basename

Syntax:

$(basename argument)

Description: Returns the filename from an argument (for example, a macro) that contains a filename with a path. For example, $(basename "/var/log/messages.log") returns messages.log. To extract the path, use the dirname template function.

Available in syslog-ng PE version 7.0.3 and later.

dirname

Syntax:

$(dirname argument)

Description: Returns the path (without the filename) from an argument (for example, a macro) that contains a filename with a path. For example, $(dirname "/var/log/messages.log") returns /var/log path. To extract the filename, use the basename template function.

Available in syslog-ng PE version 7.0.3 and later.

echo

Syntax:

$(echo argument)

Description: Returns the value of its argument. Using $(echo ${HOST}) is equivalent to ${HOST}.

env

Syntax:

$(env <environment-variable>)

Description: Returns the value of the specified environment variable. Available in syslog-ng PE 7 and later.

format-cef-extension

syslog-ng PE includes a template function (format-cef-extension) to format name-value pairs as ArcSight Common Event Format extensions. Note that the template function only formats the selected name-value pairs, it does not provide any mapping. There is no special support for creating the prefix part of a Common Event Format (CEF) message. Note that the order of the elements is random. For details on the CEF extension escaping rules format, see the ArcSight Common Event Format.

You can use the value-pairs that syslog-ng PE stores about the log message as CEF fields. Using value-pairs, you can:

  • select which value-pairs to use as CEF fields,

  • add custom value-pairs as CEF fields,

  • rename value-pairs, and so on.

For details, see Section 2.12, Structuring macros, metadata, and other value-pairs. Note that the syntax of format-* template functions is different from the syntax of value-pairs(): these template functions use a syntax similar to command lines.

Using the format-cef-extension template function, has the following prerequisites:

  • Load the the cef module in your configuration:

    @module cef
  • Set the on-error global option to drop-property, otherwise if the name of a name-value pair includes an invalid character, syslog-ng PE drops the entire message. (Key name in CEF extensions can contain only the A-Z, a-z and 0-9 characters.)

    options {
       on-error("drop-property");
    };
  • The log messages must be encoded in UTF-8. Use the encoding() option or the validate-utf8 flag in the message source.

Example 11.4. Using the format-cef-extension template function

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).

$(format-cef-extension --scope syslog,all_macros,selected_macros \
  --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
  --pair MSGHDR=\"$PROGRAM[$PID]: \")

The following example selects every value-pair that has a name beginning with .cef., but removes the .cef. prefix from the key names.

template("$(format-cef-extension --subkeys .cef.)\n")

The following example shows how to use this template function to store log messages in CEF format:

destination d_cef_extension {
  file("/var/log/messages.cef" template("${ISODATE} ${HOST} $(format-cef-extension --scope selected_macros --scope nv_pairs)\n"));
};

format-cim

Syntax:

$(format-cim)

Description: Formats the message into Splunk Common Information Model (CIM) format. Applications that can receive messages in CIM format include Kibana, logstash, and Splunk. Applications that can be configured to log into CIM format include nflog and the Suricata IDS engine.

destination d_cim {
    network("192.168.1.1" template("$(format-cim)\n"));
};

You can find the exact source of this template function in the syslog-ng PE GitHub repository.

format-json

Syntax:

$(format-json parameters)

Description: The format-json template function receives value-pairs as parameters and converts them into JavaScript Object Notation (JSON) format. Including the template function in a message template allows you to store selected information about a log message (that is, its content, macros, or other metadata) in JSON format. Note that the input log message does not have to be in JSON format to use format-json, you can reformat any incoming message as JSON.

You can use the value-pairs that syslog-ng PE stores about the log message as JSON fields. Using value-pairs, you can:

  • select which value-pairs to use as JSON fields,

  • add custom value-pairs as JSON fields,

  • rename value-pairs, and so on.

For details, see Section 2.12, Structuring macros, metadata, and other value-pairs. Note that the syntax of format-json is different from the syntax of value-pairs(): format-json uses a syntax similar to command lines.

Note

By default, syslog-ng PE handles every message field as a string. For details on how to send selected fields as other types of data (for example, handle the PID as a number), see Section 2.12.1, Specifying data types in value-pairs.

Example 11.5. Using the format-json template function

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).

$(format-json --scope syslog,all_macros,selected_macros \
  --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
  --pair MSGHDR=\"$PROGRAM[$PID]: \")

The following example shows how to use this template function to store log messages in JSON format:

destination d_json {
    file("/var/log/messages.json" template("$(format-json --scope selected_macros --scope nv_pairs)\n"));
};
Note

In case of syslog-ng macros starting with a dot (for example ".SDATA.meta.sequenceID") an empty key name is added at the top level of the JSON structure. You can work around this by adding --shift 1 as a parameter to the template function. For example in case of ".SDATA.meta.sequenceID", an empty key name is added at the top level of the JSON structure:

{"":
    {"SDATA" :
        {"meta" :
            {"sequenceID": "123"}
        }
    }
}

format-welf

This template function converts value-pairs into the WebTrends Enhanced Log file Format (WELF). The WELF format is a comma-separated list of name=value elements. Note that the order of the elements is random. If the value contains whitespace, it is enclosed in double-quotes, for example, name="value". For details on the WELF format, see https://www3.trustwave.com/support/kb/article.aspx?id=10899.

To select which value-pairs to convert, use the command-line syntax of the value-pairs() option. For details on selecting value-pairs, see Section value-pairs().

Example 11.6. Using the format-welf() template function

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).

$(format-welf --scope syslog,all_macros,selected_macros \
  --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
  --pair MSGHDR=\"$PROGRAM[$PID]: \")

The following example shows how to use this template function to store log messages in WELF format:

destination d_welf {
    file("/var/log/messages.welf" template("$(format-welf --scope selected_macros --scope nv_pairs)\n"));
};

geoip2

Syntax:

$(geoip2 --database <path-to-geoip2-database-file>
    [ --field "registered_country.names.ru" ] ${HOST})

Description: This template function exctracts specific fields from the mmdb database using the --field parameter. If you omit this parameter, the it returns the 2-letter country code of any IPv4/IPv6 address or host.

To retrieve additional GeoIP information, see Section 15.2, Looking up GeoIP2 data from IP addresses.

graphite-output

Syntax:

$(graphite-output parameters)

Description: Available in syslog-ng PE 7.0 and later. This template function converts value-pairs from the incoming message to the Graphite plain text protocol format. It is ideal to use with the messages generated by the monitor-source plugin (currently available in the syslog-ng incubator project).

For details on selecting value-pairs in syslog-ng PE and for possibilities to specify which information to convert to Graphite plain text protocol format, see Section 2.12, Structuring macros, metadata, and other value-pairs. Note that the syntax of graphite-output is different from the syntax of value-pairs(): graphite-output uses a the command-line syntax used in the format-json template function.

Example 11.7. Using the graphite-output template function

The following configuration example shows, how to send value-pairs with names starting with "vmstat." to Graphite running on localhost, port 2003:

destination d_graphite {
    network( host("localhost") port(2003) template("$(graphite-output --key vmstat.*)"));
};

grep

Syntax:

$(grep condition value-to-select)

Description: The grep template function can search a message context when correlating messages (for example, when you use a pattern database or the grouping-by parser). The context-lookup template function requires a condition (a filter or a string), and returns a specific macro or template of the matching message (for example, the ${MESSAGE} field of the message).

Example 11.8. Using the grep template function

The following example selects the message of the context that has a username name-value pair with the root value, and returns the value of the auth_method name-value pair.

$(grep ("${username}" == "root") ${auth_method})

You can to specify multiple name-value pairs as parameters, separated with commas. If multiple messages match the condition of grep, these will be returned also separated by commas. This can be used for example to collect the e-mail recipients from postfix messages.

hash

Syntax:

$(<method> [opts] $arg1 $arg2 $arg3...)

Options:

--length N, -l N

Truncate the hash to the first N characters.

Description: Calculates a hash of the string or macro received as argument using the specified hashing method. If you specify multiple arguments, effectively you receive the hash of the first argument salted with the subsequent arguments.

<method> can be one of md5, md4, sha1, sha256, sha512 and "hash", which is equivalent to md5. Macros are expected as arguments, and they are concatenated without the use of additional characters.

This template function can be used for anonymizing sensitive parts of the log message (for example username) that were parsed out using PatternDB before storing or forwarding the message. This way, the ability of correlating messages along this value is retained.

Also, using this template, quasi-unique IDs can be generated for data, using the --length option. This way, IDs will be shorter than a regular hash, but there is a very small possibility of them not being as unique as a non-truncated hash.

Example 11.9. Using the $(hash) template function

The following example calculates the SHA1 hash of the hostname of the message:

$(sha1 $HOST)

The following example calculates the SHA256 hash of the hostname, using the salted string to salt the result:

$(sha1 $HOST salted)

To use shorter hashes, set the --length:

$(sha1 --length 6 $HOST)

To replace the hostname with its hash, use a rewrite rule:

rewrite r_rewrite_hostname{set("$(sha1 $HOST)", value("HOST"));};
Example 11.10. Anonymizing IP addresses

The following example replaces every IPv4 address in the MESSAGE part with its SHA-1 hash:

rewrite pseudonymize_ip_addresses_in_message {subst ("((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])[.]){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))", "$(sha1)", value("MSG"));};

if

Syntax:

$(if (<condition>) <true template> <false template>)

Description: Returns the value of the <true template> parameter if the <condition> is true. If the <condition> is false, the value of <false template> is returned.

Example 11.11. Using pattern databases and the if template function

The following example returns violation if the username name-value pair of a message is root, and system otherwise.

$(if ("${username}" == "root") "violation" "system")

This can be used to set the class of a message in pattern database rules based on the condition.

<value name="username">$(if ("${username}" == "root") "violation" "system")</value>

Since template functions can be embedded into each other, it is possible to use another template function as the template of the first one. For example, the following expression returns root if the username is root, admin if the username is joe, and normal user otherwise.

<value name="username">
    $(if ("${username}" == "root")
        "root"
        $(if ("${username}" == "joe") "admin" "normal user"))</value>

indent-multi-line

Syntax:

$(indent-multi-line parameter)

Description: This template function makes it possible to write multi-line log messages into a file. The first line is written like a regular message, subsequent lines are indented with a tab, in compliance with RFC822.

Example 11.12. Using the indent-multi-line template function

The following example writes multi-line messages into a text file.

destination d_file {
        file ("/var/log/messages"
                template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n") );
};

ipv4-to-int

Syntax:

$(ipv4-to-int parameter)

Description: Converts the specified IPv4 address to its numeric representation. The numerical value of an IPv4 address is calculated by treating the IP address as a 4-byte hexadecimal value. For example, the 192.168.1.1 address equals to: 192=C0, 168=A8, 1=01, 1=01, or C0A80101, which is 3232235777 in decimal representation.

Note

This template function is available only if the convertfuncs module has been loaded.

By default, syslog-ng PE loads every available module. For details, see Section 5.8.1, Loading modules

length

Syntax:

$(length "<macro>")

Description: Returns the length of the macro in characters, for example, the length of the message. For example, the following filter selects messages that are shorter than 16 characters:

f_short {
    match ('-', value ("$(if ($(length "${MESSAGE}") <= 16) "-" "+")"));
};

lowercase

Syntax:

$(lowercase "<macro>")

Description: Returns the lowercase version of the specified string or macro. For example, the following example uses the lowercase version of the hostname in a directory name:

destination d_file {
    file ("/var/log/${MONTH}/${DAY}/$(lowercase "${HOST}")/messages");
};

Available in syslog-ng PE 7.0 and later.

Numerical operations

Syntax:

$(<operation> "<value1>" "<value2>")

Description: These template functions allow you to manipulate numbers, that is, to perform addition (+), substraction (-), multiplication (*), division (/), and modulus (%). All of them require two numeric arguments. The result is NaN (Not-a-Number) if the parameters are not numbers, cannot be parsed, or if a division by zero would occur. For example, to add the value of two macros, use the following template function:

$(+ "${<MACRO1>}" "${<MACRO2>}");

When you are correlating messages and a name-value pair contains numerical values in the messages, you can calculate the lowest (min), highest (max), total (sum), and mean (average) values. These calculations process every message of the correlation context. For details on message correlation, see Chapter 14, Correlating log messages. For example, if the messages of the context have a .myfields.load name-value pair, you can find the highest load value using the following template function.

$(max ${.myfields.load})

or

Syntax:

$(or <macro1> <macro2>)

Description: This template function returns the first non-empty argument.

padding

Syntax:

 $(padding <macro> <width> <prepended-character-or-string>)

Description: This template function returns the value of its first parameter (a string or macro), prepended with a string. This string is <width> long, and repeats the character or string set in the third parameter. If you use a single character, it is added <width> times. If you use a string, it is repeated until its length reaches <width>. The default padding character is ' ' (space). For example:

Example 11.13. Using the padding template function

If the value of the ${MESSAGE} macro is mymessage, then the output of the padding() template function is the following:

$(padding ${MESSAGE} 10 X)

Output: XXXXXXXXXXmymessage

$(padding ${MESSAGE} 10 foo)

Output: foofoofoofmymessage

python

Syntax:

 $(python <name-of-the-python-method-to-use> <arguments-of-the-method>)

Description: This template function enables you to write a custom template function in Python. You can define a Python block in your syslog-ng PE configuration file, and define one or more Python functions in it. You can use these functions as template functions. If you use a Python block, syslog-ng PE embeds a Python interpreter to process the messages. Note the following points:

  • Available in syslog-ng PE version 7.0.2 and later.

  • Currently only Python 2.7 is supported.

    Warning

    If you are planning to use Python in syslog-ng PE (for example Python parser or Python template function) on RHEL 6 platform, then you have to manually install Python 2.7. If the Python version on the machine is not 2.7, you will receive a similar error message during startup:

    [2017-07-27T13:42:03.606679]
    Reading shared object for a candidate module; path='/opt/syslog-ng/lib/syslog-ng', fname='mod-python.so', module='mod-python'
    [2017-07-27T13:42:03.606994]
    Error opening plugin module; module='mod-python', error='libpython2.7.so.1.0: cannot open shared object file: No such file or directory'
  • The Python block must be a top-level block in the syslog-ng PE configuration file. If you store the Python code in a separate Python file and only include it in the syslog-ng PE configuration file, make sure that the PYTHON_PATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example: export PYTHONPATH=/opt/syslog-ng/etc

  • The Python object is initiated only once, when syslog-ng PE is started or reloaded. That means it keeps the state of internal variables while syslog-ng PE is running.

  • The Python block can contain multiple Python functions.

  • Using Python code in syslog-ng PE can significantly decrease the performance of syslog-ng PE, especially if the Python code is slow.

  • Validate and lint the Python code before using it. The syslog-ng PE application does not do any of this.

  • Support disclaimer: 

    Using Python in syslog-ng PE is recommended only if you are familiar with both Python and syslog-ng PE. Product support applies only to syslog-ng PE: that is, until the entry point of the Python code and passing the specified arguments to the Python code. Balabit is not responsible for the quality, resource requirements, or any bugs in the Python code, nor any syslog-ng PE crashes, message losses, or any other damage caused by the improper use of this feature, unless explicitly stated in a contract with Balabit.

  • The first argument in the definition of the Python function is the actual log message. This is implicitly passed to the function, you do not have to use it in the template function.

  • The value of the template function is return value of the Python function.

  • To reference a name-value pair or a macro in the Python code, use the following format. For example, if the first argument in the definition of the function is called log-message, the value of the HOST macro is log-message['HOST'], and so on. (The log-message contains the entire log message (not just the text body) in a structure similar to a Python dict, but it is actually an object.)

  • You can define new name-value pairs in the Python function. For example, if the first argument in the definition of the function is called log-message, you can create a new name-value pair like this: log_message["new-macro-name"]="value". This is useful when you parse a part of the message from Python, or lookup a value based on data extracted from the log message.

    Note that the names of the name-value pairs are case-sensitive. If you create a new name-value pair called new-macro-name in Python, and want to reference it in another part of the syslog-ng PE configuration file (for example, in a template), use the ${new-macro-name} macro.

  • You cannot override hard macros (see Section 11.1.4, Hard vs. soft macros).

  • To list all available keys (names of name-value pairs), use the log_message.keys() function.

Declaration: 

python {
    def <name_of_the_python_function>(<log_message>, <optional_other_arguments>):
        # <your-python-code>
        return <value_of_the_template_function>
};

template <template-name> {
    template($(python <name_of_the_python_function>));
};
Example 11.14. Writing template functions in Python

The following example creates a Python template function called return_message that returns the MESSAGE part of the log message.

@version: 7.0.8

python {
    def return_message(log_message):
        return log_message['MESSAGE']
};

destination d_local {
    file("/tmp/logs.txt" template("[$(python return_message)]\n"));
};

The following example creates a Python template function called resolve_host that receives an IP address as an argument, and attempts to resolve it into a hostname.

@version: 7.0.8

python {
    import socket

    def resolve_host(log_message, hostname):
        try:
            return socket.gethostbyaddr(hostname)[0]
        except (socket.herror, socket.error):
            return 'unknown'
};

destination d_local {
    file("/tmp/logs.txt" template("${ISODATE} $(python resolve_host(${SOURCE_IP})) ${MESSAGE}\n"));
};

replace-delimiter

Syntax:

$(replace-delimiter "<old-delimiters>" "<new-delimiter>" "<macro>")

Description: Replaces the delimiter character with a new one. For example, the following example replaces the tabulators (\t) in the message with semicolons (;):

$(replace-delimiter "\t" ";" "${MESSAGE}")

Available in syslog-ng PE 7.0 and later.

sanitize

Syntax:

$(sanitize <options> "<macro1>" "<macro2> ...")

Description: This file replaces the special characters in macro values, for example, it can replace the slash (/) characters in a filename with the underscore (_) character. If you specify multiple arguments, they will be concatenated using the / character, so they can be used as separate directory levels when used in filenames.

The function has the following options:

--ctrl-chars or -c

Filter control characters (characters that have an ASCII code of 32 or lower). This option is used by default.

--invalid-chars <characterlist> or -i <characterlist>

The list of characters to be replaced with underscores (_). The default list contains the / character. The following example replaces the \ and @ characters, so for example, fo\o@bar becomes foobar:

$(sanitize -i \@ $PROGRAM)
--no-ctrl-chars or -C

Do not filter the control characters (characters that have an ASCII code of 32 or lower).

--replacement <replacement-character> or -r <replacement-character>

The character used to replace invalid characters. By default, this is the underscore (_). The following example replaces invalid characters with colons instead of underscores, so for example, foo/bar becomes foo;bar:

$(sanitize -r ; $PROGRAM)
Example 11.15. Using the sanitize template function

The following example uses the sanitize function on two macros, and the results are used as directory names in a file destination.

file("/var/log/$(sanitize $HOST $PROGRAM)/messages");

This is equivalent to file("/var/log/$HOST/$PROGRAM/messages");, but any slashes in the values of the $HOST and $PROGRAM macros are replaced with underscores.

strip

Syntax:

$(strip "<macro>")

Description: Deletes whitespaces from the beginning and the end of a macro. You can specify multiple macros separated with whitespace in a single template function, for example:

$(strip "${MESSAGE}" "${PROGRAM}")

substr

Syntax:

$(substr "<argument>" "<offset>" "<length>")

Description: This function extracts a substring of a string.

argument

The string to extract the substring from, for example, "${MESSAGE}"

offset

Specifies where the substring begins (in characters). 0 means to start from the beginning of the string, 5 means to skip the first 5 characters of the string, and so on. Use negative numbers to specify where to start from the end of the string, for example, -1 means the last character, -5 means to start five characters before the end of the string.

length

Optional parameter: The number of characters to extract. If not specified, the substring will be extracted from the offset to the end of the string. Use negative numbers to stop the substring before the end of the string, for example, -5 means the substring ends five characters before the end of the string.

Example 11.16. Using the substr template function

Skip the first 15 characters of the message, and select the rest:

$(substr "${MESSAGE}" "15");

Select characters 16-30 of the message (15 characters with offset 15):

$(substr "${MESSAGE}" "15" "15");

Select the last 15 characters of the message:

$(substr "${MESSAGE}" "-15");

A template that converts the message to RFC3164 (BSD-syslog) format and truncates the messages to 1023 characters:

template t_truncate_messages {
    template("$(substr \"<$PRI>$DATE $HOST $MSGHDR$MESSAGE\" \"0\" \"1023\")\n");
    template-escape(no);
};

uppercase

Syntax:

$(uppercase "<macro>")

Description: Returns the uppercase version of the specified string or macro. For example, the following example uses the uppercase version of the hostname in a directory name:

destination d_file {
file ("/var/log/${MONTH}/${DAY}/$(uppercase "${HOST}")/messages");
};

Available in syslog-ng PE 7.0 and later.

uuid

Syntax:

$(uuid)

Description: Generates a Universally Unique IDentifier (UUID) that complies with RFC4122. That way, an UUID can be added to the message soon after it is received, so messages stored in multiple destinations can be identified. For example, when storing messages in a database and also in files, the UUID can be used to find a particular message both in the database and the files.

To generate a UUID, you can use a rewrite rule to create a new value-pair for the message.

Example 11.17. Using Universally Unique Identifiers

The following example adds a value-pair called MESSAGE_UUID to the message using a rewrite rule and a template.

rewrite r_add_uuid { set("$(uuid)" value("MESSAGE_UUID")); };

destination d_file {
    file ("/var/log/messages"
          template("$MESSAGE_UUID $ISODATE $HOST $MSG\n")
          template-escape(no)
         );
};

log { source(s_network);
      rewrite(r_add_uuid);
      destination(d_file);
};
Note

This template function is available only if the tfuuid module has been loaded.

By default, syslog-ng PE loads every available module. For details, see Section 5.8.1, Loading modules