Working with parsed Active Roles logs in syslog-ng

In my previous Active Roles blog, you learned how to forward Active Roles logs to a central syslog-ng server to parse and store the logs. In this blog, I’ll show you how to:

- Work with parsed Active Roles logs.

- Store logs to various document stores.

- Prepare long-term storage.

- Send alerts for some critical events.

Creating a log management layer using syslog-ng PE can save you valuable resources. This is because unlike in environments that use several SIEMs and log analytics software, in case of syslog-ng, you need to collect log messages only once, parse them at the central location, and then forward only relevant logs to each destination. This way, you can save not just valuable network and storage resources, but also licensing costs, as most SIEMs are licensed based on log volume.

Before you begin

If you want to test the configuration described in this blog, you need 2-4 hosts. Installing Active Roles needs 1-3 Windows Server machines, while the syslog-ng PE server requires a supported Linux host. Describing the system requirements or the installation processes are outside the scope of this blog. Contact One Identity at https://www.syslog-ng.com/register/115582/ to receive a trial version of syslog-ng Premium Edition and help to get started. For Active Roles, you can reach One Identity at https://www.oneidentity.com/register/62175/.

Name-value pairs

Name-value pairs (sometimes also called “macros”) are probably one of the most useful parts of syslog-ng. Macros have been a core feature of syslog-ng since the beginning: incoming log messages were parsed by syslog-ng and various fields (priority, facility, date, program, PID and message) were stored into name-value pairs. You could use these to reformat log messages or use these values in file names.

Without additional parsing, the message part of a log message is treated by syslog-ng as one long string. While this might be valid in some use cases, many log messages look like an almost complete English sentence with some variable parts in it. Just think about SSH login messages: they include the user name, the source IP and port, and the login method embedded in a sentence. You might want to create an alert in syslog-ng for such messages (for example, if a root user logs in). However, if the whole MESSAGE is a single string, you cannot do that.

PatternDB can find important information in unstructured log messages, like the above-mentioned SSH login message, and create name-value pairs from the information it finds. There are various parsers for structured log messages as well, like the CSV, JSON and XML parsers. You can also combine these and build some really complex parsers. Sometimes, the syslog header might be missing from log messages, so you need to create a parser for the whole message.

Message parsing and name-value pairs give you a lot more flexibility when it comes to filtering or templating log messages. For example, you can send an alert to Slack when someone logged in as a root user through SSH, or you can forward two important fields from an extremely long log message to save network bandwidth and storage.

Configuring the syslog-ng PE server

In my previous blog, we built a syslog-ng configuration which collected log messages from Windows using the RFC5424 syslog protocol, parsed the messages using the XML parser, and saved the incoming logs both as-is and in JSON format, so that we could see the name-value pairs parsed from XML. We build on that configuration and extend it in several ways by utilizing the name-value pairs parsed from the log messages. Namely, we will add four new destinations:

- An Elasticsearch destination, often used for short-term storage for easy searching and reporting.

- An OpenObserve destination as an example for specifying additional destinations.

- A LogStore destination, providing secure long-term storage for log messages.

- And a Slack destination for alerting when someone deletes an object from AD using Active Roles.

You should append the following configuration to your syslog-ng.conf:

# source for Windows clients, RFC5424
source s_win {
  syslog(port(601));
};

# xml parser for Windows XML logs
parser p_xml {
  xml(prefix('winxml.'));
};

# destination for Windows logs
destination d_fromwin {
  file("/var/log/fromwin");
  file("/var/log/fromwin.json" template("$(format-flat-json --scope rfc5424
        --scope dot-nv-pairs --rekey .* --shift 1
        --scope nv-pairs --exclude MESSAGE)\n") );
};

# Elasticsearch destination for easy searching and reporting
destination d_elasticsearch_http {
    elasticsearch-http(
        index("syslog-ng")
        type("")
        user("elastic")
        password("-w35jWfY_syp577m-UfS")
        url("https://localhost:9200/_bulk")
        template("$(format-json –omit-empty-values
        --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE --exclude MESSAGE
        --exclude winxml.Event.EventData.*
        @timestamp=${ISODATE})")
        tls(peer-verify(no))
    );
};

# OpenObserve destination
destination d_openobserve_http {
    elasticsearch-http(
        index("syslog-ng")
        type("")
        user("peter@czanik.hu")
        password("-w35jWfY_syp577m-UfS")
        url("http://127.0.0.1:5080/api/default/_bulk")
        template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE --exclude MESSAGE
        --exclude winxml.Event.EventData.*
        --key ISODATE @timestamp=${ISODATE})")
    );
};

# long term storage
destination d_logstore {
  logstore("/var/log/ars.lgs" compress(9) );
};

# alerts to file and Slack
destination d_alerts {
  file("/var/log/alerts.txt");
  slack(
    hook-url("https://hooks.slack.com/services/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")
    template("User ${.SDATA.win@18372.4.EVENT_USERNAME} deleted a ${.SDATA.win@18372.4.EVENT_SID_TYPE} on ${HOST} using ARS")
  );
};

# log path for Windows logs
log {
  source(s_win);
  if ("${PROGRAM}" ne "syslog-ng-agent") {
    parser(p_xml);
  };
  destination(d_fromwin);
  destination(d_elasticsearch_http);
  destination(d_openobserve_http);
  if ("${PROGRAM}" eq "ARAdminSvc") {
    destination(d_logstore);
  };
  if ("${.SDATA.win@18372.4.EVENT_TASK}" eq "ObjectDelete") {
    destination(d_alerts);
  };
};

Let’s check this configuration in detail!

The source s_win is a network source using the RFC5424 syslog protocol to collect log messages from syslog-ng Agent for Windows. For simplicity, no TLS is configured in this example. However, for a production environment, using encryption is of course highly recommended for network traffic.

The next building block in the configuration is an XML parser. The syslog-ng Windows Agent is configured to send Windows event logs in XML format, as this way, it is easy to turn log messages into name-value pairs.

The destination called d_fromwin actually includes two file destination drivers. The first one simply writes the incoming log into a file, while the other one writes the name-value pairs parsed from the XML logs into JSON format. You might not need these in the final configuration, but they are very useful while building your syslog-ng configuration. Compared to the previous blog, we did a minor change here: we replaced format-json with format-flat-json. While format-json shows the structure of the log message better, format-flat-json shows the names we can use in the syslog-ng configuration as names of the name-value pairs.

The next building block in the configuration is an Elasticsearch destination. Here, the interesting part is the template. We had an introduction to name-value pairs earlier, so here I want to focus on the format-json template function. Its parameters define which name-value pairs are included in the JSON and how. The –scope option helps to select groups of name-value pairs. We remove the leading dots from the names, as those are normally replaced by underscores by syslog-ng, but that has a special meaning when sent to Elasticsearch. We exclude the date, as we include it in a different format. The original XML message is not included, as it is redundant information. EventData is also excluded, as it might contain information in a format that cannot be ingested by Elasticsearch.

Note that removing the leading dot in the JSON template does not actually change the name of the name-value pairs. It only changes how the name appears in the JSON output. I often forget about this myself and wonder why my template utilizing the name without the leading dot does not work… :-)

The OpenObserve destination is included because it is often asked whether syslog-ng can send logs to multiple destinations at the same time. Yes, it can, and OpenObserve is becoming a popular destination among syslog-ng users.

Logstore is the secure long-term storage format of syslog-ng. In this configuration, we only enable log compression. Depending on the input data and the level of compression, it allows you to drastically reduce the space needed to store your log messages. However, Logstore also allows you to encrypt and time stamp log messages using an external Time Stamping Authority. This way, you can prove that a log message arrived at a specific time and is unmodified.

The next destination in the configuration is for alerts. The file destination was added just for testing and can be safely deleted once everything works as expected. The actual alert that will be used will send the log message to Slack using a template. Names from the flat JSON file came in handy while building this template.

Finally, the log statement is the heart of the configuration. This is what connects the previously described building blocks together, deciding how logs are processed and where they are stored. Logs are collected from the syslog source, and then parsed by the XML parser, unless the application name is syslog-ng-agent.

The next three lines in the log statement make sure that all logs are saved to files, Elasticsearch and OpenObserve. Then, all logs from Active Roles are saved to a logstore. Finally, if someone deletes an object from AD using Active Roles, syslog-ng sends an alert to Slack.

What is next?

Even if this configuration looks complex, it is simplified in many ways. For example, as I already mentioned, you should use encryption in a production environment, but for testing purposes, we simply send messages in clear text in this case. If you looked at the logs in Elasticsearch carefully, you might have also noticed that syslog-ng Agent for Windows messages are empty. We discard the MESSAGE macro in the template, as forwarding it properly would add even more complexity. Of course, you would need to address this for a production environment.