Alerting on One Identity Cloud PAM Essentials logs using syslog-ng

One Identity Cloud PAM Essentials is the latest security product by One Identity. It provides asset management as well as secure and monitored remote access for One Identity Cloud users to hosts on their local network. I had a chance to test PAM Essentials while still in development. While there, I also integrated it with syslog-ng.

From my previous blog, you could learn what PAM Essentials is, and how you can collect its logs using syslog-ng. This blog will show you how to work with the collected log messages and create alerts when somebody connects to a host on your local network using PAM Essentials.

Before you begin

This is a technical blog about syslog-ng. I show you enough PAM Essentials to understand what I am doing on the syslog-ng side. If you want to learn more about One Identity Cloud PAM Essentials, check the press release at: https://www.oneidentity.com/community/news/b/press-releases/posts/one-identity-cloud-pam-essentialssm-launched-to-streamline-management-and-strengthen-protection-of-privileged-access .

On the syslog-ng side you need the syslog-ng Windows Agent (which is a commercial product) to collect the logs of the One Identity Network Agent. On the server side you can use both syslog-ng Premium Edition or Open Source Edition. The configuration I show works with both. In this case I used PE 7.0.34 (the currently available latest version), and the same configuration should work with any recent 3.X OSE version. You might need some minor changes for syslog-ng 4.X due to type support.

What we try to achieve

Before going into the syslog-ng configuration details, let me summarize again, what we try to achieve with using syslog-ng:

• Collecting all logs related to PAM Essentials generated on the local network to a single location. This includes One Identity Network Agent logs, the Windows Eventlog of the host, where the software runs, and also the hosts where outside users connect through PAM Essentials.

• Parsing Windows XML logs, so it is turned into easy-to-use name-value pairs.

• Parsing One Identity Network Agent logs to find connection information.

• Storing all incoming log messages into an Elasticsearch database, where logs can easily be searched.

• Sending alerts to Slack instant messaging, once a connection is established through PAM Essentials.

And a few words about what we do not try to achieve. We collect Windows Eventlog from the host running the One Identity Network Agent, and all logs from the Linux hosts running the ssh servers. We even parse the XML-formatted logs to turn them into name-value pairs. However, we do not work with those logs any further (at least, not in this blog), just store them into local text files and into Elasticsearch. They are still useful: check what I wrote about central logging in my previous blog.

Forwarding logs from your ssh (Linux) host

We want to collect all PAM Essentials related log messages to a central location. It means that we also have to install syslog-ng on the Linux host(s) running the ssh server, where the outside user connects using PAM Essentials. The following configuration snippet works both with syslog-ng OSE and PE:

destination d_net {
  tcp("172.16.167.170" port(514));
};
log {
  source(s_sys);
  destination(d_net);
};

Note that the source name for local logs might be different on your system. Of course, the IP address of the syslog-ng server on your network is different, and on a production system you might want to filter what to forward, use disk-buffer, encryption, and so on. For testing, the above configuration is good enough.

Forwarding One Identity Network Agent and Windows eventlog

In my previous blog (URL TBD) I showed you how to configure the syslog-ng Windows Agent to forward Windows and One Identity Network Agent logs. Check that blog for information how to configure the syslog-ng Windows Agent.

Configuring the syslog-ng server for log collection and initial message parsing

The following configuration collects log messages both from Linux and Windows. It parses the Windows eventlog messages using an XML parser. In the end is saves all logs locally, and also to Elasticsearch.

# source for Linux clients, RFC3164
source s_lin {
  tcp(port(514));
};

# source for Windows clients, RFC5424
source s_win {
  syslog(port(601));
};

# xml parser for Windows XML logs
parser p_xml {
  xml(prefix('winxml.'));
};

# destination for Linux logs
destination d_fromlin {
  file("/var/log/fromlin");
  file("/var/log/fromlin.json" template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs)\n") );
};

# Elasticsearch destination
destination d_elasticsearch {
  elasticsearch-http(
      index("syslog-ng")
      type("")
      user("elastic")
      password("Do2oFMbINS3hzfmR8uIV")
      url("https://172.16.167.182:9200/_bulk")
      template("$(format-json --scope rfc5424 --scope dot-nv-pairs
      --rekey .* --shift 1 --scope nv-pairs
      --exclude DATE @timestamp=${ISODATE})")
      tls(peer-verify(no))
  );
};

# destination for Windows logs
destination d_fromwin {
  file("/var/log/fromwin");
  file("/var/log/fromwin.json" template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs)\n") );
};

# log path for Linux logs
log {
  source(s_lin);
  destination(d_fromlin);
  destination(d_elasticsearch);
};

# log path for Windows logs
log {
  source(s_win);
# syslog-ng-agent logs do not come in XML format
# only parse the rest
  if ("${PROGRAM}" ne "syslog-ng-agent") {
    parser(p_xml);
  };
  destination(d_fromwin);
  destination(d_elasticsearch);
};

The Windows-related part should be mostly familiar from my previous blog. The only change there is the addition of an Elasticsearch destination. Why is that important? It makes viewing and searching name-value pairs parsed from Windows Eventlog a lot more easy. Yes, JSON-formatted text files are essential when you build a syslog-ng configuration, but a GUI is a lot more convenient for production use.

So, what is new here compared to my previous blog? Other than the previously mentioned Elasticsearch destination, the collection of Linux logs. There is a tcp() source, and a file destination, which writes both a traditional looking syslog file and JSON formatted messages. In this case JSON formatting does not make much sense, I am just too much used to creating it while working on configurations.

Both log path utilizes the same elasticsearch-http() destination, as it provides a single, easy to use web interface to search all the logs we collect.

Parsing connection logs

Now that we have all logs related to PAM Essentials generated on the local network collected to a single location, we can add one more thing to our configuration: alerting to slack when someone connects to a host on our local network using PAM Essentials.

First, let us take a look at the relevant log messages from the One Identity Network Agent log file, log.txt. The first one is generated when the user is connected, the second one when the connection is closed.

c:\program files\one identity\network agent\logs\log.txt: 2024-02-29 12:40:55.627 +01:00 [DBG] [secure-gateway-client:err] [2024-02-29T11:40:55Z DEBUG client::connection] Initiating new connection; client: 60032829-c4f5-44eb-892c-916651f807c9; id: 55d75647-032e-4955-9e08-2654c50b4e9c; target: 172.16.167.182:22
c:\program files\one identity\network agent\logs\log.txt: 2024-02-29 16:52:31.187 +01:00 [DBG] [secure-gateway-client:err] [2024-02-29T15:52:31Z DEBUG client::connection] Connection closed; id: 18caa31e-692e-4168-8d64-b285df952c22

Actually, the above two lines are the content of MESSAGE macro in the message that the syslog-ng Windows Agent forwards. The original message is slightly shorter: the file name is added to the message by the syslog-ng Windows Agent. The content of this macro is what we have to parse using PatternDB.

PatternDB is a parser in syslog-ng, which can parse unstructured log messages and turn relevant information into name-value pairs. These name-value pairs can be used for filtering, or stored as name-value pairs for easier search. Of course syslog-ng does not know anything about the content of log messages by itself. PatternDB needs an XML database that describes the log messagges, and using that database syslog-ng can turn relevant information from the log messages into name-value pairs.

Explaining PatternDB from scratch is not the scope of this blog. If you are new to PatternDB, you should read the documentation at: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.38/administration-guide/91#TOPIC-2026478

You can find the XML file describing the One Identity Network Agent connection logs below. Save it as oina.pdb (you can use any other name and extension, but this is the file name the syslog-ng.conf in this blog expects).

<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='3' pub_date='2024-02-29'>
  <ruleset name='syslog-ng-agent' id='c800f393-45c3-456b-8c8e-e685ea947a36'>
    <description>
      This ruleset covers the One Identity Network Agent logs as forwarded by PE Windows Agent.
    </description>
    <url>www.oneidentity.com</url>
    <pattern>syslog-ng-agent</pattern>
    <rules>

      <rule provider="CzP" id="86b694bf-868d-45d8-8347-a7ef8d72f4c2" class="system">
        <patterns>
          <pattern>@ESTRING:oi.filename:: @@ESTRING:oi.date: [@@ESTRING:oi.level:]@@ESTRING::client: @@ESTRING:oi.clientid:; id: @@ESTRING:oi.sessionid:; target: @@ESTRING:oi.targethost::@@ANYSTRING:oi.targetport@</pattern>
        </patterns>
        <examples>
          <example>
           <test_message program="syslog-ng-agent">c:\program files\one identity\network agent\logs\log.txt: 2024-02-29 12:40:55.627 +01:00 [DBG] [secure-gateway-client:err] [2024-02-29T11:40:55Z DEBUG client::connection] Initiating new connection; client: 60032829-c4f5-44eb-892c-916651f807c9; id: 55d75647-032e-4955-9e08-2654c50b4e9c; target: 172.16.167.182:22</test_message>
           <test_values>
            <test_value name="oi.filename">c:\program files\one identity\network agent\logs\log.txt</test_value>
            <test_value name="oi.date">2024-02-29 12:40:55.627 +01:00</test_value>
            <test_value name="oi.level">DBG</test_value>
            <test_value name="oi.clientid">60032829-c4f5-44eb-892c-916651f807c9</test_value>
            <test_value name="oi.sessionid">55d75647-032e-4955-9e08-2654c50b4e9c</test_value>
            <test_value name="oi.targethost">172.16.167.182</test_value>
           </test_values>
          </example>
        </examples>
      </rule>

      <rule provider="CzP" id="3e9bdb97-c15c-4707-a461-21cfe60a145e" class="system">
        <patterns>
          <pattern>@ESTRING:oi.filename:: @@ESTRING:oi.date: [@@ESTRING:oi.level:]@@ESTRING::closed; id: @@ANYSTRING:oi.sessionid@</pattern>
        </patterns>
        <examples>
          <example>
           <test_message program="syslog-ng-agent">c:\program files\one identity\network agent\logs\log.txt: 2024-02-29 16:52:31.187 +01:00 [DBG] [secure-gateway-client:err] [2024-02-29T15:52:31Z DEBUG client::connection] Connection closed; id: 18caa31e-692e-4168-8d64-b285df952c22</test_message>
           <test_values>
            <test_value name="oi.filename">c:\program files\one identity\network agent\logs\log.txt</test_value>
            <test_value name="oi.date">2024-02-29 16:52:31.187 +01:00</test_value>
            <test_value name="oi.level">DBG</test_value>
            <test_value name="oi.sessionid">18caa31e-692e-4168-8d64-b285df952c22</test_value>
           </test_values>
          </example>
        </examples>
      </rule>


    </rules>
  </ruleset>
</patterndb>

I do not want to go in depth how the PatternDB XML works, here I just share some of the most important information with you. Each rule in the database has pattern that describes the log message: the constant parts, the variable parts, how to find them in the log message and what to name the found information. This is followed by a test message and test values for the extracted values.

For us the most important information is the list of name-value pairs extracted from the log messages. In this case:

• oi.filename: the name of the log file

• oi.date: the date as written by the One Identity Network Agent

• oi.level: the log level set by the One Identity Network Agent

• oi.clientid: the unique identifier of the One Identity Network Agent

• oi.sessionid: the unique identifier for the session (network connection) through PAM Essentials

• oi.targethost: the host on the internal network, where the outside user connects through PAM Essentials

In this blog we use the oi.targethost name-value pair. If this exists in a log message, it means that a new connection is established. In this case we save the event in a separate file, and also send an alert to Slack.

So, what do we need to add to the previous configuration?

• The PatternDB parser with the above XML file.

# patterndb parser for OI Network Agent connection logs
parser p_connect {
  db-parser(file("/opt/syslog-ng/etc/oina.pdb"));
};

• A Slack destination (or Telegram, etc.). You can learn more about the Slack destination from one of my previous blogs: https://www.syslog-ng.com/community/b/blog/posts/send-your-log-messages-to-slack

# Slack destination
destination d_slack {
  slack(
    hook-url("https://hooks.slack.com/services/T06xxxDFE/B06xxxJLA/gywxxxUdpmJOC")
    template("OI Network Agent connection at ${DATE} to ${oi.targethost}")
  );
};

• Add the parser and the destination to the log path for Windows logs.

# log path for Windows logs
log {
  source(s_win);
# syslog-ng-agent logs do not come in XML format
# only parse the rest
  if ("${PROGRAM}" ne "syslog-ng-agent") {
    parser(p_xml);
  };
  parser(p_connect);
# send alert if there is a connection through the OI Network Agent
# this macro is only created from connection logs
  if ("${oi.targethost}" ne "") {
    destination {file("/var/log/alert" template("${DATE} OI Network Agent connection to ${oi.targethost}\n"));};
    destination(d_slack);
  };
  destination(d_fromwin);
  destination(d_elasticsearch);
};

The final configurations

For your copy & paste convenience I put together everything below. You should append it to syslog-ng.conf, or if your syslog-ng.conf supports it, create a new configuration snippet for it under the /etc/syslog-ng/conf.d/ directory.

######
# PAM Essentials

# source for Linux clients, RFC3164
source s_lin {
  tcp(port(514));
};

# source for Windows clients, RFC5424
source s_win {
  syslog(port(601));
};

# xml parser for Windows XML logs
parser p_xml {
  xml(prefix('winxml.'));
};

# patterndb parser for OI Network Agent connection logs
parser p_connect {
  db-parser(file("/opt/syslog-ng/etc/oina.pdb"));
};

# destination for Linux logs
destination d_fromlin {
  file("/var/log/fromlin");
  file("/var/log/fromlin.json" template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs)\n") );
};

# Elasticsearch destination
destination d_elasticsearch {
  elasticsearch-http(
      index("syslog-ng")
      type("")
      user("elastic")
      password("Do2oFMbINS3hzfmR8uIV")
      url("https://172.16.167.182:9200/_bulk")
      template("$(format-json --scope rfc5424 --scope dot-nv-pairs
      --rekey .* --shift 1 --scope nv-pairs
      --exclude DATE @timestamp=${ISODATE})")
      tls(peer-verify(no))
  );
};

# destination for Windows logs
destination d_fromwin {
  file("/var/log/fromwin");
  file("/var/log/fromwin.json" template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs)\n") );
};

# Slack destination
destination d_slack {
  slack(
    hook-url("https://hooks.slack.com/services/T06PxxxFE/B06xxxA/gywQkG8XXXmJOC")
    template("OI Network Agent connection at ${DATE} to ${oi.targethost}")
  );
};

# log path for Linux logs
log {
  source(s_lin);
  destination(d_fromlin);
  destination(d_elasticsearch);
};

# log path for Windows logs
log {
  source(s_win);
# syslog-ng-agent logs do not come in XML format
# only parse the rest
  if ("${PROGRAM}" ne "syslog-ng-agent") {
    parser(p_xml);
  };
  parser(p_connect);
# send alert if there is a connection through the OI Network Agent
# this macro is only created from connection logs
  if ("${oi.targethost}" ne "") {
    destination {file("/var/log/alert" template("${DATE} OI Network Agent connection to ${oi.targethost}\n"));};
    destination(d_slack);
  };
  destination(d_fromwin);
  destination(d_elasticsearch);
};

Testing

Once you saved and reloaded the syslog-ng configuration, you are ready for some testing. Luckily testing is now easy. All you have to do is to connect to a machine on your local network through PAM Essentials and check your logs. The above configuration saves alerts to a text file too, so you can check alerting even without configuring Slack or any other instant messaging service for syslog-ng previously.

You should see similar messages in /var/log/alert:

Mar 12 16:37:08 OI Network Agent connection to 127.80.65.77
Mar 12 16:37:19 OI Network Agent connection to 127.80.65.77
Mar 12 16:38:10 OI Network Agent connection to 172.16.167.182

You can find a screenshot from Slack below:

You can see that there is a target IP address from the internal network. However, there are some random looking IP addresses from the localhost: 127.80.65.77 It is used for management, and accessible only from localhost. Fun fact: the numbers are not random.

• 127 – local host

• 80 – ASCII ‘P’

• 65 – ASCII ‘A’

• 77 – ASCII ‘M

You can either live with the extra alerts, or modify the syslog-ng configuration to filter those out.

What is next?

Being able to search all PAM Essentials related log messages in a single place and receiving a real-time alert on instant messaging when a user logs in through PAM Essentials are both valuable additions to your PAM Essentials installation. You can further improve your configuration with a bit of correlation. This way you could also receive an alert when a session is closed without needing to check the PAM Essentials dashboard regularly for new session recordings to appear. You can gain some inspiration on implementation from one of my recent blogs: https://www.syslog-ng.com/community/b/blog/posts/aggregating-messages-in-syslog-ng-using-grouping-by Where I aggregated ssh login / logout messages.

-

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.