3 Mar 2021

When to use the syslog-ng agent for Windows?

You can collect log messages from a Windows host in multiple ways using syslog-ng. For large scale installations the easiest is to use the Windows Event Collector (WEC) component of syslog-ng Premium Edition (PE). This way you don’t have to install any new client software on the Windows side, just point the WEC to the destionation to send their log messages.

Please note that WEC only works for Windows EventLog. If you need to collect log messages from text files, you need to install the syslog-ng agent for Windows on your hosts. For example, web servers often log to files instead of Windows EventLog.

Let’s review how to do a standalone installation of syslog-ng agent for Windows and then see the differences between using the legacy (RFC3164) and the new (RFC5424) syslog protocol.

Before you begin

The syslog-ng agent for Windows is commercial software, for the purposes of this test you can use the trial version available after registration. There is no syslog-ng server for Windows, so you also need a Linux host with syslog-ng to collect log messages from your Windows hosts.

For this blog post I downloaded “syslog-ng Agent with MMC snap-in” from the trial site. This is one of the ways to install syslog-ng, but this one is the best suited for a standalone installation. The MMC snap-in makes sure that you can configure syslog-ng agent for Windows using a graphical user interface. Other options (not covered here) include an XML-based configuration file and configuring the agent through Windows Group Policies.

On the server side I used syslog-ng Open Source Edition (OSE), as I did not use any of the extra features of PE, like using the Advanced Log Transfer Protocol (ALTP) for message transfer. Installation is not covered here, only the configuration part.

To generate some log files, I installed the Apache HTTP server on the Windows host, but you can use anything that generates text-based log messages.

Installing the agent

Once you downloaded the latest version of the syslog-ng agent for Windows, the standalone installation is a next, next, finish experience. At the end the installer automatically starts the configuration interface. You can safely close it for now, we’ll come back to it later.

Once you finished installing the syslog-ng agent for Windows, you will find a new “syslog-ng agent for Windows” menu under the Start menu with three application shortcuts and links to the documentation. You will use these to configure the syslog-ng agent for Windows and to restart it.

Configuring the syslog-ng server

The syslog-ng PE server runs on Linux or AIX. The Open Source Edition (OSE) also runs on FreeBSD. No matter what you use, the following simple configuration works with any recent syslog-ng release. You can either append it to syslog-ng.conf, or drop a new configuration in the /etc/syslog-ng/conf.d/ directory, if syslog-ng on your host is configured to use it.

source s_windows {
  tcp(port(514));
};
destination d_fromwindows {
  file("/var/log/fromwindows");
};
log {
  source(s_windows);
  destination(d_fromwindows);
};

source s_windowsnew {
  syslog(port(601));
};
destination d_windowsjson {
  file("/var/log/windowsjson" template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs)\n\n"));
};
log {
  source(s_windowsnew);
  destination(d_windowsjson);
};

This configuration defines two sets of sources and destinations. The first one uses the legacy syslog protocol on port 514. The second one uses the new syslog protocol on port 601. In this case the destination file uses JSON formatting to show the name-value-pairs created by the syslog-ng agent for Windows and forwarded using the new syslog protocol.

Don’t forget to reload syslog-ng for the new configuration to take effect.

Configuring syslog-ng agent for Windows

Choose “Configure syslog-ng agent for Windows” from the Start menu. Once it started click on the “syslog-ng agent settings” on the left-hand panel, then on “local settings” and finally on “destinations”. You will see some filtering related settings below it. For now, we leave those alone.

To add your first destination right click on “destinations”. It brings up a small menu, where choose “add new server”. It opens a new window. On the first tab enter the name or IP address of your server and click “OK”.

To add your second destination right click on “destinations” again and choose “add new server”. This time after configuring the server name or address, change the port to 514 and then switch to “messages” tab at the top of the window. From the drop-down list at the top choose “legacy BSD syslog protocol” and click “OK”.

Now stop and start syslog-ng agent for Windows using the menu items in the Windows Start menu for the configuration to take effect.

Checking your logs

With both the syslog-ng server and the agent for Windows configured you are now ready to check your log messages. While the agent is configured to use the new syslog protocol by default, most people still stick to the legacy protocol. Compare the two log files to see the difference:

==> /var/log/fromwindows <==
Mar  2 09:09:52 172.16.167.130 Microsoft_Windows_security_auditing.[3140]: DESKTOP-228O3BU\Peter Czanik: Security Microsoft Windows security auditing.: [Success Audit] A logon was attempted using explicit credentials.    Subject:   Security ID:  NT AUTHORITY\SYSTEM   Account Name:  DESKTOP-228O3BU$   Account Domain:  WORKGROUP   Logon ID:  0x3E7   Logon GUID:  {00000000-0000-0000-0000-000000000000}    Account Whose Credentials Were Used:   Account Name:  Peter Czanik   Account Domain:  DESKTOP-228O3BU   Logon GUID:  {00000000-0000-0000-0000-000000000000}    Target Server:   Target Server Name: localhost   Additional Information: localhost    Process Information:   Process ID:  0x594   Process Name:  C:\Windows\System32\svchost.exe    Network Information:   Network Address: 127.0.0.1   Port:   0    This event is generated when a process attempts to log on an account by explicitly specifying that account’s credentials.  This most commonly occurs in batch-type configurations such as scheduled tasks, or when using the RUNAS command. (EventID 4648)
==> /var/log/windowsjson <==
{"SOURCE":"s_windowsnew","SDATA":{"win@18372.4":{"EVENT_USERNAME":"DESKTOP-228O3BU\\Peter Czanik","EVENT_TYPE":"Success Audit","EVENT_TASK":"Logon","EVENT_SOURCE":"Microsoft Windows security auditing.","EVENT_SID":"N/A","EVENT_REC_NUM":"31227","EVENT_PROVIDER":"Microsoft-Windows-Security-Auditing","EVENT_NAME":"Security","EVENT_LEVEL":"0","EVENT_ID":"4648","EVENT_HOST":"DESKTOP-228O3BU","EVENT_FACILITY":"16","EVENT_CATEGORY":"Logon"},"meta":{"sysUpTime":"-30095","sequenceId":"994"}},"PROGRAM":"Microsoft_Windows_security_auditing.","PRIORITY":"notice","PID":"3140","MESSAGE":"DESKTOP-228O3BU\\Peter Czanik: Security Microsoft Windows security auditing.: [Success Audit] A logon was attempted using explicit credentials.\r\n\r\nSubject:\r\n Security ID:  NT AUTHORITY\\SYSTEM\r\n Account Name:  DESKTOP-228O3BU$\r\n Account Domain:  WORKGROUP\r\n Logon ID:  0x3E7\r\n Logon GUID:  {00000000-0000-0000-0000-000000000000}\r\n\r\nAccount Whose Credentials Were Used:\r\n Account Name:  Peter Czanik\r\n Account Domain:  DESKTOP-228O3BU\r\n Logon GUID:  {00000000-0000-0000-0000-000000000000}\r\n\r\nTarget Server:\r\n Target Server Name: localhost\r\n Additional Information: localhost\r\n\r\nProcess Information:\r\n Process ID:  0x594\r\n Process Name:  C:\\Windows\\System32\\svchost.exe\r\n\r\nNetwork Information:\r\n Network Address: 127.0.0.1\r\n Port:   0\r\n\r\nThis event is generated when a process attempts to log on an account by explicitly specifying that account’s credentials.  This most commonly occurs in batch-type configurations such as scheduled tasks, or when using the RUNAS command. (EventID 4648)","HOST_FROM":"172.16.167.130","HOST":"172.16.167.130","FACILITY":"local0","DATE":"Mar  2 09:09:52"}

Unless your only reason to collect logs from Windows hosts is to tick a check box on a compliance check list, using the legacy syslog protocol does not make much sense. Even if the logs include a good part of the event data finding that in free-form text messages is difficult and creating reports from these logs is close to impossible. Using the new syslog protocol provides you with richer data, and what is even more important: structured data.

In the server side configuration we simply dump incoming name-value pairs in JSON format into a file. But even with this raw data you should see that alerting and reporting is a lot easier when you have structured data at hand. You do not have to parse the logs yourself but use the name-value pairs directly in your filters for alerting or log routing. And if you forward name-value pairs instead of plain text to NoSQL databases, like Elasticsearch, you can easily query your logs and create reports.

Forwarding logs from files

The distinctive feature of syslog-ng agent for Windows is that it can also forward log messages from text files. For example, web servers tend to log to text files even on Windows. In my examples I use logs from XAMPP. But any other text-based log file will work for testing purposes.

On Windows choose “Configure syslog-ng agent for Windows” from the start menu. On the left side panel click “syslog-ng agent settings” and “local settings”. Right click on “file sources” and choose “properties”. Enable file sources and click “OK”.

You should now switch to the right hand panel. Click on “file sources”. You can now add a file source. You can configure now which file(s) to read. Choose a directory, enter a file name (or * for all files) and configure an application name, for example “apache”.

Confirm the changes and restart the syslog-ng agent for Windows for the settings to take effect. You should now see the logs on your syslog-ng server:

leap152:~ # tail  /var/log/fromwindows /var/log/windowsjson
==> /var/log/fromwindows <==
Mar  2 11:08:26 172.16.167.130 apache[3140]: c:\xampp\apache\logs\access.log: 127.0.0.1 - - [02/Mar/2021:11:06:31 +0100] "GET /dashboard/javascripts/all.js HTTP/1.1" 200 188385 "http://127.0.0.1/dashboard/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36"

==> /var/log/windowsjson <==
{"SOURCE":"s_windowsnew","SDATA":{"meta":{"sysUpTime":"-177751","sequenceId":"1102"},"file@18372.4":{"size":"30556","position":"29666","name":"c:\\xampp\\apache\\logs\\access.log"}},"PROGRAM":"apache","PRIORITY":"info","PID":"3140","MESSAGE":"c:\\xampp\\apache\\logs\\access.log: 127.0.0.1 - - [02/Mar/2021:11:06:32 +0100] \"GET /dashboard/images/favicon.png HTTP/1.1\" 200 2508 \"http://127.0.0.1/dashboard/\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36\"","HOST_FROM":"172.16.167.130","HOST":"172.16.167.130","FACILITY":"local0","DATE":"Mar  2 11:08:26"}

As you can see, the JSON version contains some useful extra information again.

Windows Agent vs WEC – how to pick?

As you can see, the syslog-ng agent for Windows can solve all your logging tasks on Windows. Still, the introduction suggested to use WEC whenever possible. If you need to collect logs from text-based log files, you have no choice, you have to use syslog-ng agent for Windows. In all other cases WEC is easier. You do not have to install any applications on the Windows side and WEC uses structured logging all the time.

When to use the syslog-ng agent for Windows?