Talking to users at both IT security and operations related conferences, they confessed the same: the less software the better. Less software to learn, maintain and follow in bug trackers. The Windows Event Collector (WEC) component of syslog-ng Premium Edition (PE) brings this to the extremes: it enables Windows machines to act as log sources without installing any software. Just enable the WEC component of syslog-ng PE on your Linux server and enjoy the logs flowing in.
From this blog you can learn how to set up a test environment for WEC. I will also show you an easier way to prepare the certificates used, help you avoid some of the failures, and point you to alternative ways, where necessary. I’ll try to refer to the excellent syslog-ng PE WEC documentation whenever possible.
And before you start installing WEC, I also want to make sure that you know its limitations. You can only collect Windows EventLog type log messages using WEC. Some applications, typically web servers, save their log files to text files. If you need to collect those centrally, WEC cannot help you – you will need to install the syslog-ng agent for Windows on these hosts.
Before you begin
The syslog-ng PE WEC is commercial software, the trial version is available from One Identity. WEC is a component of the syslog-ng PE server and thus runs on Linux. For the purposes of this test you will also need at least one Windows host.
While for this blog I installed syslog-ng PE on a CentOS 7 host and used Windows Server 2019 as Windows host, any Linux release supported by syslog-ng PE should work, just as any Windows version. The WEC feature we use was introduced way back in Windows Vista, so any Microsoft-supported Windows version will work.
Installing WEC
WEC is a component of syslog-ng PE and is installed as part of syslog-ng PE. However, it’s not started by default, as it needs some configuration before it can start accepting those Windows logs. You need to configure at least the certificates before you can use it.
So, what is the syslog-ng WEC? WEC is not a syslog-ng module, but a separate application. It acts as a gateway between Windows hosts and syslog-ng PE. It collects the forwarded Windows EventLog messages from Windows hosts over TLS encrypted HTTP connections and sends them through a socket to syslog-ng PE.
Creating certificates
The most error-prone part of configuring WEC is creating the certificates. syslog-ng WEC and Windows hosts authenticate each other using certificates. Windows is very picky about certificates, and prefers well configured DNS to function properly. In order to work around this problem and avoid hours of painful debugging, here we show an IP-based configuration. Both the certificates and the configurations use only IP addresses and no host names or FQDN.
The WEC documentation contains examples how to create certificates with multiple host names and IP addresses. Note, those examples only work with properly working DNS servers on your network.
We create the certificates on the Linux host running syslog-ng PE. Most Linux distributions have OpenSSL installed by default. We use OpenSSL from the command line to generate a certificate authority (CA) and keys for the server (WEC) and clients (Windows hosts).
Start by creating a fresh new directory and changing to it. For the purpose of this blog, I used /root/cert2/, but it could be anywhere on the file system available by the time WEC starts.
Create a new configuration file with the name certopts.cnf with the following two lines as content:
[client_server_auth] extendedKeyUsage = serverAuth,clientAuth
The name of the file or the name between the square brackets could be anything, but the examples below expect that you use these names. We will use it later when creating server and client certificates.
As a first step we create our very own certificate authority. As it is a test certificate in my home lab, I filled it out with some random data. The different fields mean:
- “C” stands for country
- “L” for location, usually city name
- “ST” for state (we do not use that here in Hungary)
- “O” organization
- “OU” organizational unit
- “CN” stands for common name. In case of CA it is the name shown, when you list certificate authorities. In case of client or server certificates it is the FQDN of the host or the IP address.
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -days 3650 -out ca.crt -subj '/C=HU/L=Budapest/O=My Home Ltd/OU=Operations/CN=Operations Root CA'
After creating the CA create a fingerprint of it. We will use it while configuring the Windows part.
openssl x509 -in ca.crt -fingerprint -sha1 -noout | sed -e 's/\://g' > fingerprint.txt
While creating the server keys, make sure that you use the IP address of the host running syslog-ng PE as common name. In my case it is 172.16.167.137.
openssl req -new -newkey rsa:4096 -nodes -out server.csr -keyout server.key -subj '/C=HU/L=Budapest/O=My Home Ltd/OU=Operations/CN=172.16.167.137'
openssl x509 -req -in server.csr -out server.crt -CA ca.crt -CAkey ca.key -CAcreateserial -extfile certopts.cnf -extensions client_server_auth -days 365
Instructions for the client are similar, just the file names and the common name is changed:
openssl req -new -newkey rsa:4096 -nodes -out client.csr -keyout client.key -subj '/C=HU/L=Budapest/O=My Home Ltd/OU=Operations/CN=172.16.167.50'
openssl x509 -req -in client.csr -out client.crt -CA ca.crt -CAkey ca.key -CAcreateserial -extfile certopts.cnf -extensions client_server_auth -days 365
As a final step, export the client key in a format Windows can import:
openssl pkcs12 -export -inkey client.key -in client.crt -certfile ca.crt -out client.p12
Configuring Windows
The Windows part of the configuration cannot be simplified compared to what is in the documentation. So, instead of reproducing here what is there, I ask you to read the relevant section. Here I just add some extra information.
The certificates were generated on a Linux host. You need client.p12 and fingerprint.txt on your Windows host. Of course you can e-mail the certificate or share them over SMB, but there is an easier solution if you have an ssh server running on Linux. You can use scp on Windows to copy the files.
The documentation shows how to add the NETWORK SERVICE account to the Event Log Readers group. The way described works in most cases, except when the Windows host is a domain controller (like my test machine was). Instead of “Local Users and Groups” (which does not work on a domain controller host) you have to use Active Directory Users and Computers (ADUC). Select the name of your domain, then “Builtin”. You will find the “Event Log Reader” group on the right-side panel, where you can do the change.
When adding the subscription in SubscriptionManagers dialog, make sure that you use the exact same name, or in our case IP address as the common name in the server certificate. Also, open fingerprint.txt in a text editor, so that you can copy & paste the hash to the dialog.
Configuring WEC
Before you can use WEC you have to configure it. It does not work out of box, as you have to configure at least the server name (or in our case, the IP address) and the location of certificates. You can find the WEC configuration in /opt/syslog-ng/etc/wec.yaml and as you could gess from the extension of the file, its format is YAML. Make sure that the server field contains the exact same FQDN, or in our case IP address, as the common name in the server certificate. Point the keyfile, certfile, cadir fields to the right locations. You can leave the rest of the configuration as is. In my case the configuration is:
server: "172.16.167.137"
port: 5986
keyfile: "/root/cert2/server.key"
certfile: "/root/cert2/server.crt"
cadir: "/root/cert2/"
log:
level: "info"
file: "/opt/syslog-ng/var/wec.log"
eventdestination:
# file: "events.log"
unixdatagram: "/opt/syslog-ng/var/run/wec.sock"
subscriptions:
- name: "ExampleDefault"
computers:
- "*"
contentformat: "RenderedText"
heartbeats: 900.000
connectionretry: 60.0
batchtimeoutlimit: 900.000
queries: |
<QueryList>
<Query Id="0">
<Select Path="Application">*</Select>
<Select Path="Security">*</Select>
<Select Path="Setup">*</Select>
<Select Path="System">*</Select>
<Select Path="ForwardedEvents">*</Select>
</Query>
</QueryList>
Configuring syslog-ng
On the syslog-ng side you need a source, which collects log messages from WEC. It is very simple:
source s_wec {
windowsevent();
};
By default this parses all messages coming from WEC and saves the resulting name-value pairs with a “.windowsevent.” prefix. You can change the prefix using the prefix() option.
Of course you can simply save the message part of log messages, but one of the main advantage of using the WEC component of syslog-ng that it collects structured log messages from Windows hosts. You can see all the collected name-value pairs if you store the logs for example in JSON format and the template includes name-value pairs.
Here is the complete configuration to append to your syslog-ng.conf:
source s_wec {
windowsevent();
};
log {
source(s_wec);
destination {
file("/var/log/example.log"
template("$(format-json --scope dot-nv-pairs)\n")
);
};
};
Note, that the template mentions dot-nv-pairs. It is due to the leading dot in the default prefix. It works here, but if you want to store your logs to Elasticsearch, make sure that you use a prefix without a leading dot.
Reload syslog-ng for the configuration to take effect:
systemctl reload syslog-ng
Testing WEC from the command line
With everything in place, you can now test WEC from the command line:
/opt/syslog-ng/sbin/wec -c /opt/syslog-ng/etc/wec.yaml -s /opt/syslog-ng/var/wecstate/
You should see a few startup messages:
2021-03-25T15:31:28.941+0100 INFO Event collector server started {"port": 5986}
2021-03-25T15:31:28.941+0100 INFO Trying to connect to unix datagram socket {"unix-datagram": "/opt/syslog-ng/var/run/wec.sock"}
2021-03-25T15:31:28.943+0100 INFO Connected to unix datagram socket {"unix-datagram": "/opt/syslog-ng/var/run/wec.sock"}
2021-03-25T15:31:28.944+0100 INFO New CA cert {"filename": "/root/cert2/ca.crt", "blockIndex": 0, "RootCA": true, "_ca_thumbprint": "42B17EAF954B0B889D483C40FB91E1620735BA35"}
And maybe a few warnings, as the CA directory in the configuration also contains unrelated files. After a few minutes you should see logs arriving in /var/log/example.log. You can speed up this process by executing the following commands in a cmd window on your Windows host:
net stop winrm
net start winrm
You can stop WEC by pressing ^C. To start WEC as a service, use:
systemctl start syslog-ng-wec
Starting WEC by default
Until now WEC was not started by default on the system. But once you verified that everything works as expected, you should start it automatically, just like syslog-ng PE. The following command enables the WEC service:
systemctl enable syslog-ng-wec
Next time you boot your system, WEC will start automatically.
Troubleshooting
If you do not see incoming log messages even after a few minutes, it is time to do some troubleshooting. Multiple things can go wrong, here we just give a few starting points where to check.
First, check if you can reach WEC from your Windows host. You can use telnet to port 5986 of the Linux host from Windows. If you see an error message related to unsuccessful TLS connection in the WEC console output, then you have no firewall problems.
Double check that the URL and the hash are spelled correctly in the subscription dialog on Windows.
You can also check EventLog on the Windows host. Just don’t have high expectations, as the error messages are long, without correctly describing the problem. You can find related log messages both under “Eventlog-ForwardingPlugin” and “Windows Remote Management” as well.
What is next?
This quick tutorial walked you through your first WEC installation. Now that it works, time to scale it. You can register to our webinar showcasing that use case: scaling WEC with clustering, using syslog-ng. Register here.
