Windows Event Collector clustering with Syslog-ng

Hello, and thanks for joining me here today at Virtual Unite. My name is James Bonamico, and I'm a Senior Sales Engineer here at One Identity. And I'll be talking about a new feature that's soon to make its way into Syslog-ng. And that is Windows Event Collector Clustering. We'll go through how to collect events from your entire Windows infrastructure at enterprise scales agentlessly and with multiple node redundancy. For this presentation, I'll assume you already have some familiarity with Syslog-ng itself as well as Windows events. First, let me back up a bit to go over the current ways you can use Syslog-ng to capture logs from Windows servers and desktops. First of all, we have the Syslog-ng Windows Agent, which is included with Syslog-ng Premium Edition as well as the Syslog-ng store box. So this is a lightweight agent that can be installed in standalone mode or pushed out through a group policy. It's easily configured through an MMC either locally or centrally. And what it does is allow you to capture Windows Event Container Logs and stream them over to a Syslog-ng server. Not only that, but it can also monitor files and folders as well. So if you have an application that logs to flat files, these can also be easily read and ingested. There are robust built-in filtering and formatting options available so you can customize what data to send out over the wire. And it can all be encrypted during that transit. So this slide here, what you're looking at is a basic diagram on how you would use the Agent to collect logs from a Windows environment. On the left, you have your various data centers and offices where the agent has been deployed, and they're all streaming logs to Syslog-ng Premium Edition. Here we're using a Load Balancer in the middle there so we can have multiple Syslog-ng nodes for redundancy. And then on the Syslog-ng PE servers themselves, we can parse, filter, reformat, and route those messages to any number of final destinations. So whether you're sending the Splunk, QRadar, Elastic Kafka, or cloud destinations like Google Pub/Sub, Stackdriver, or Azure Sentinel, we've got you covered. And then next up is this is Syslog-ng Windows Event Collector, or WEC. So this approach, on the other hand, is agentless. So if you can't install anything, there is really no need to install into your Windows environment for it to work. It's based on Windows event forwarding, so it's a proven Microsoft technology. It allows you to define source-initiated push subscriptions and have them forwarded over HTTPS using TLS mutually authenticated encryption. So here is how that works. On the Syslog-ng server, and running alongside Syslog-ng itself, is this Syslog-ng WEC process. Subscriptions are configured here in a YAML file. So once Windows authenticates the Syslog-ng server, the WEC service then lets Windows know which event containers it's interested in, so Windows can then begin sending those events. The WEC service receives these events and then passes them to a Syslog-ng Windows Events source, which then runs them through an XML Parser to break out all key-value pairs like application, event ID, et cetera. Then from there, Syslog-ng can filter, process, and reformat the logs however you want before storing or sending them on to a SIEM or other log consumer. So just to be clear here, the Syslog-ng Windows Event Collector and the Syslog-ng server-- they are two processes running side by side in the Syslog-ng server itself. So these aren't actually two separate servers. And then to scale either of these methods up, whether agent or agentless, you can use Syslog-ng Relays. So relays are full Syslog-ng instances whose purpose is to aggregate and, if necessary, pre-filter and pre-process, and then pass those logs onto a Syslog-ng server. It's the same software, but the difference is that the relay is intended as a pass-through. It's not a destination in itself. And it will always work with a server instance. So you could have these relays acting as a local ingest point for a few thousand endpoints each, and then ensure a secure and reliable communication from those relays to a central server. So let me pause the slides for a minute here, and I'll switch to my terminal. What I'm going to show you is some of the internals of the Windows Event Collector, really essentially what it looks like. So this here is the YAML file that defines the subscription. So Windows will have a list of one or more collectors that it will contact and provide logs to through [? WIN or M ?] according to the subscriptions defined in this YAML file. So you can see here, this is the level of events that we want. It's informational and above. So we're subscribed to the application, the security, the setup system, and forwarded events containers. The rest of the stuff here is just connection parameters. And also, we're not doing any event ID filtering in this particular case, but it is possible in this YAML file. And then down here, what we're looking at are the state files that WEC generates. So you can see, an entry will be created in this folder for every single client, one separate file per client. And then down at the bottom here, if we look into the actual contents of one of these files-- for example, we can see that it says from this particular client, here is a list of bookmarks. And these bookmarks tell you where we are at, reading each of the event channels that we're subscribed to. So the event channel last left off at security at this record number, set up container at this record number, and then at the end it shows which channel we last received. So, currently reading from the system event channel. So this all works fantastically, but there's one problem. And I'll ask you if you can spot it. So