syslog-ng and Microsoft Sentinel

Hello, and thank you for joining me for this segment at Virtual Unite. My name is James Bonamico, and I'm a senior sales engineer here at One Identity. So today we're going to talk about one of the newer custom destination drivers that's now available in syslog-ng premium edition. This allows you to take logs that have been processed by syslog-ng and send them by API to an Azure Sentinel workspace. So as you know, syslog-ng is a universal enterprise log management solution designed from the bottom up for performance, scalability, and flexibility. It is truly the foundation of log management when it comes to being able to ingest nearly any type of log, from any platform, over any protocol, and parse, filter, process, and route them at unmatched speed and scale. And by utilizing these capabilities to filter out less important logs and normalize critical parsed data into those formats required by SIEM systems in other log consumers, you're reducing complexity and increasing the efficiency of those other systems-- and ultimately, that results in the reduction of IT expenditure. So part of the value of syslog-ng are all out of the box connectivity options. The latest push has been to accommodate the explosive growth of cloud platforms, so we've implemented several new cloud connectors-- such as Google Stackdriver, Google Pub/Sub, Microsoft Office 365, and of course, Azure Sentinel. So what you see here is a very high-level architecture overview of syslog-ng deployment environment. So on the left, you have your various log sources-- such as cloud services, applications, security and network devices, of course, servers. And they'll be sending logs in all different types of formats over various protocols-- like UDP, TCP, encrypted via TLS-- into syslog-ng. In many cases, you want to have a source like energy relay as the first ingest point. Now, if you have a relay installed at, say, at every data center, or a typical site, those relays are close to the sources, and they can ingest reliably. They can serve as aggregators. They can pre-process those logs if necessary, pre-filter them before sending them on to syslog-ng's server. And then that connection between the relay and server can be reliable as well as encrypted. And then from there, syslog-ng server will then further parse and process and filter that data, and then send it out to the various log consumers, wherever you need those logs. Maybe those logs need to go into multiple different destinations, in different formats. Well, syslog-ng can do that. So whether you're writing to file, writing into a database-- Elastic, Kafka, Splunk, Hadoop, Google Cloud, and then, of course, Azure Sentinel-- those options, and others, are available to you. So to get started with Sentinel, the first thing you need to do is create a log analytics workspace. Once you do this, you'll need to take note of two pieces of information. So that's the workspace ID and the primary key. With these credentials, we'll be able to connect the syslog-ng Sentinel destination driver with Azure Sentinel itself. And to find these pieces of information, you can go to your workspace, then under Settings, you'll see all the Agent managements tab. However, with syslog-ng, we won't actually be needing Sentinel agents at all, since we'll be sending logs directly via the API. So that's one less thing you'll need to install and manage. But here's where the keys are listed, and can be regenerated if need be. So with this information from Microsoft in hand, it's time to edit the syslog-ng configuration file. So here you can see that the basic configuration is extremely simple. You just name the destination, declare the Sentinel destination driver, and then plug in the workspace ID and auth secret that you've obtained from Azure. This is all you need to then create a log path and start sending logs from syslog-ng into Sentinel. However, there are a few other options available if you'd like to customize the data or connection, and I'll go over some of them here. So first are the batching parameters, which are disabled by default. These all tell syslog-ng when to fire off a post command to the Sentinel API. If not specified, we'll send each log message as it comes in, but for efficiency and speed, you can tune this behavior. You can set the total amount of data accumulated as the threshold. So for example, we'll send a bunch of messages in one go every, say, five megabytes. Or you can do the same when a certain number of lines are reached, like 10 or 100. Lastly, the batch timeout regulates when we'll send a batch after there's a break in the incoming stream, and the unsent data doesn't yet reach either of the other thresholds. So for example, 100 messages come in, but then there's no activity on the endpoint for a few minutes. We'll wait until the batch timeout to send any unsent messages, regardless of if they fill a batch or not. So now for the actual data. So the fields parameter is the basic set of key value pairs included in the payload by default. If not specified, this is what is sent, but again, you don't need to add this line if you're not changing it. So you can customize this to include or exclude any set of elements. So you can see, we have a few pieces of information here. You can assign the macros however you like, but these are the defaults. And then syslog-ng also gives you the ability to override the body of the message itself. Again, here are the defaults, which simply take all the key value elements in the fields parameter and arrange them in a JSON data structure. So it's JSON that we're actually sending into Azure. And then, here are some examples of the various ways you can customize this. You can add scopes, which in syslog-ng energy are predefined sets of structured data