One Identity Cloud PAM Essentials is the latest security product by One Identity. It provides asset management as well as secure and monitored remote access for One Identity Cloud users to hosts on their local network. I had a chance to test PAM Essentials while still in development. While there, I also integrated it with syslog-ng.
From this blog, you can learn what PAM Essentials is, and how you can collect its logs using syslog-ng. My next blog will show you how to work with the collected log messages and create alerts when somebody connects to a host on your local network using PAM Essentials.
Before you begin
This is a technical blog about syslog-ng. I will give you enough PAM Essentials background information so you can understand what I am doing on the syslog-ng side. If you want to learn more about PAM Essentials, check the press release at: https://www.oneidentity.com/community/news/b/press-releases/posts/one-identity-cloud-pam-essentialssm-launched-to-streamline-management-and-strengthen-protection-of-privileged-access .
On the syslog-ng side, you need syslog-ng Windows Agent (which is a commercial product) to collect the logs of the One Identity Network Agent (more about it later). On the server side, you can use both syslog-ng Premium Edition or Open Source Edition. The configuration I show you works with both. In this case, I used PE 7.0.34 (the currently available latest version), and the same configuration should work with any recent 3.X OSE version. You might need some minor changes for syslog-ng 4.X due to type support.
What is One Identity Cloud PAM Essentials?
PAM Essentials is a new product in the One Identity product family. It provides asset management and secure and monitored remote access for One Identity Cloud users through the cloud to their local networks, either SSH or RDP. As a seasoned Linux admin, I tested SSH.
What does it mean for the users? They can click on a resource on the PAM Essentials User Portal, and it generates a URL for an SSH connection. If configured properly, the SSH client opens and logs in to the host automatically. By default, the sessions are recorded, so you can play them back once a session is over. The user only has to learn how to use the new web interface, but for the rest, the tools do not change.
How does it work?
PAM Essentials is a cloud service with an additional application running on your local network. This application is called One Identity Network Agent, and it runs on Windows. I tested it on Windows 11 Insider, and it worked perfectly. However, for a production environment, Windows Server is recommended. Once installed, it provides a secure connection between your local network and the One Identity cloud.
Once you have a local network added to PAM Essentials, you can onboard target Linux and Windows hosts and user accounts that remote users can connect to. From that point on, PAM Essentials will manage and rotate credentials, securing the account. Finally, PAM Essentials also connects all these building blocks by using policies.
Why syslog-ng?
Using syslog-ng, you can collect your log messages centrally. Why is central logging so important? Convenience, availability, and security. Convenience, as you have a single place to check instead of many. Availability, as you can check log messages, even if the sending host is unavailable. Security, as logs leave hosts as soon as they are produced, so an attacker has no chance to modify them. Developers, operators, and security have a single view of the whole network and can easily correlate events from multiple hosts.
In this case, you can collect One Identity Network Agent and Windows operating system logs using the syslog-ng Windows Agent and collect Linux logs from the hosts accessed through SSH using syslog-ng. All syslog-ng Windows Agents and clients forward log messages to a central syslog-ng server. As a result, you have a single place where you can view logs from any of your PAM solution related hosts.
In this blog, I show you how to collect logs using the syslog-ng Windows Agent, what defaults to change, and how to parse and store the results on the syslog-ng server on Linux.
My next blog will show you how to work with the collected log messages, store the parsed logs in Elasticsearch, and create alerts when somebody connects to a host on your local network using PAM Essentials. This way, you can get real-time alerts about connections and have a single view of any events that happened during remote access to your network.
Configuring the syslog-ng server
The syslog-ng server configuration in the first part of my blog is relatively simple. Just append it to syslog-ng.conf, or create a new config snippet under /etc/syslog-ng/conf.d/ if the syslog-ng configuration on your server uses this directory.
# source for Windows clients, RFC5424
source s_win {
syslog(port(601));
};
# xml parser for Windows XML logs
parser p_xml {
xml(prefix('winxml.'));
};
# destination for Windows logs
destination d_fromwin {
file("/var/log/fromwin");
file("/var/log/fromwin.json" template("$(format-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs)\n") );
};
# log path for Windows logs
log {
source(s_win);
# syslog-ng-agent logs do not come in XML format
# only parse the rest
if ("${PROGRAM}" ne "syslog-ng-agent") {
parser(p_xml);
};
destination(d_fromwin);
};
Obviously, my initial configuration was even shorter. Just a syslog() source, a simple file destination, and a log path connecting the two. However, using the initial configuration, a lot of information would be lost. Let me explain what we have here and why:
-
The syslog-ng Windows Agent uses the RFC5424 protocol, thus we use the syslog() source. This is not a production configuration, so in order to keep it simple, no security and reliability features are enabled.
-
Using XML formatting for Windows Eventlog, you gain many easy-to-use name-value pairs. The XML parser turns XML data into name-value pairs.
-
We have two file destinations. The first one is a regular syslog file with a date, host, application name and log message. While it’s OK not to have any name-value pairs, much of the information is lost if you forward structured data along your logs. This is why there is a second log file with JSON formatting, containing all RFC5424 fields and also name-value pairs parsed from log messages.
-
Finally, the log path connects all components from above. The if statement before the XML parser makes sure that only XML formatted logs are parsed (in order to avoid error messages). Internal logs of the Windows Agent and logs read form files, like the One Identity Network Agent logs, do not have XML formatting.
Once you saved your configuration and reloaded the syslog-ng server, you are ready to configure the Windows Agent.
Configuring the syslog-ng Windows Agent
You can do an initial configuration of your syslog-ng Windows Agent during installation. You can also do it later. Either way, make sure that the server IP address or host name is set to your syslog-ng server. On the “messages” tab, replace the Eventlog template with:
${EVENT_MSG_XML}
This ensures that Windows Eventlog is forwarded as XML instead of plain text. This way, it is easy to generate name-value pairs on the syslog-ng server from Windows Eventlog.
You should also add a file source for the One Identity Network Agent logs. The directory is:
C:\Program Files\One Identity\Network Agent\Logs
The name of the log file is log.txt, which you type in or browse. If you open the file in Notepad or a different text editor, you will see that while most of the log messages are single line, there are also some really long multi-line log messages. You can set multi-line options at the bottom of the tab. Set “Application” to custom, and enter the following regular expression into the “Prefix” field:
[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3} .[0-9]{2}:[0-9]{2} \[.{3}\]
Yes, it is ugly, but this way, multi-line log messages are forwarded as a single log message. You can learn more about multi-line prefixes, and how this regexp was created from one of my previous blogs: https://www.syslog-ng.com/community/b/blog/posts/working-with-multi-line-logs-in-syslog-ng If you take a closer look at the example logs in that blog, you will see some similarities with log.txt :-)
Restart the syslog-ng Windows Agent service for the configuration to take effect.
Testing
Once both the syslog-ng server and the Windows Agent were configured and reloaded, you should see incoming log messages in the log files. Take a look at both the plain text file and the JSON formatted file to appreciate the differences. You should see similar logs in the JSON formatted file:
One Identity Network Agent:
{"classifier":{"class":"unknown"},"SOURCE":"s_win","SDATA":{"meta":{"sysUpTime":"14332","sequenceId":"775"},"file@18372.4":{"size":"242624","position":"242624","name":"c:\\program files\\one identity\\network agent\\logs\\log.txt"}},"PROGRAM":"syslog-ng-agent","PRIORITY":"info","PID":"3996","MESSAGE":"c:\\program files\\one identity\\network agent\\logs\\log.txt: 2024-03-08 15:02:05.407 +01:00 [INF] Agent connected. Status = Connected\r","HOST_FROM":"172.16.167.128","HOST":"172.16.167.128","FACILITY":"local0","DATE":"Mar 8 15:02:16"}
Generic Windows message:
{"winxml":{"Event":{"_xmlns":"http://schemas.microsoft.com/win/2004/08/events/event","System":{"Version":"0","TimeCreated":{"_SystemTime":"2024-03-08T14:23:46.8632296Z"},"Task":"0","Provider":{"_Name":"OiNetworkAgent"},"Opcode":"0","Level":"4","Keywords":"0x80000000000000","Execution":{"_ThreadIDD":"0","_ProcessID":"2560"},"EventRecordID":"4413","EventID":{"_Qualifiers":"0","EventID":"0"},"Computer":"czp-win11vm","Channel":"Application"},"RenderingInfo":{"_Culture":"en-GB","Task":"None","Opcode":"Info","Message":"\"Service started successfully.\"","Level":"Information","Keywords":{"Keyword":"Classic"}},"EventData":{"Data":"\"Service started successfully.\""}}},"classifier":{"class":"unknown"},"SOURCE":"s_win","SDATA":{"win@18372.4":{"EVENT_TYPE":"Information","EVENT_TASK":"None","EVENT_SOURCE":"OiNetworkAgent","EVENT_SID":"N/A","EVENT_REC_NUM":"4413","EVENT_PROVIDER":"OiNetworkAgent","EVENT_NAME":"Application","EVENT_LEVEL":"4","EVENT_ID":"0","EVENT_HOST":"czp-win11vm","EVENT_FACILITY":"16","EVENT_CATEGORY":"None"},"meta":{"sysUpTime":"143326","sequenceId":"1823"}},"PROGRAM":"OiNetworkAgent","PRIORITY":"info","PID":"3996","MESSAGE":"<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='OiNetworkAgent'/><EventID Qualifiers='0'>0</EventID><Version>0</Version><Level>4</Level><Task>0</Task><Opcode>0</Opcode><Keywords>0x80000000000000</Keywords><TimeCreated SystemTime='2024-03-08T14:23:46.8632296Z'/><EventRecordID>4413</EventRecordID><Correlation/><Execution ProcessID='2560' ThreadID='0'/><Channel>Application</Channel><Computer>czp-win11vm</Computer><Security/></System><EventData><Data>Service started successfully.</Data></EventData><RenderingInfo Culture='en-GB'><Message>Service started successfully.</Message><Level>Information</Level><Task>None</Task><Opcode>Info</Opcode><Keywords><Keyword>Classic</Keyword></Keywords></RenderingInfo></Event>","HOST_FROM":"172.16.167.128","HOST":"172.16.167.128","FACILITY":"local0","DATE":"Mar 8 15:23:46"}
You can find more information about parsing Windows Eventlog XML in one of my previous blogs at: https://www.syslog-ng.com/community/b/blog/posts/dedicated-windows-xml-eventlog-parser-in-syslog-ng
What is next?
JSON formatted text files are very important while developing a new syslog-ng configuration. They are also good, as they preserve all the name-value pairs parsed from log messages. However, they are are painfully not human-readable. So, in my next blog, I’ll show you how to forward logs to Elasticsearch for easier search and visualization. You will also learn how to collect logs from the Linux server (where the remote user connected through SSH), so all your logs are stored at the same place. We will also parse connection logs from the One Identity Network Agent and create alerts when a user logs out. This way, you can watch the session recording immediately, and check what the remote user did.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.