The syslog-ng Premium Edition (PE) application is built on the solid foundation of syslog-ng Open Source Edition (OSE). PE inherits many of the OSE features and adds cloud-related features, among others. Our latest webinar shows you how to send log messages to Azure Event Hub and Google Pub/Sub.
The development of syslog-ng OSE is done in the open, you can follow it on GitHub. syslog-ng OSE has quite a few experimental features, used and thus tested only by a small group of users. PE is built on OSE, but only includes a stable and well-tested subset of OSE features. On top of the OSE features, PE also has some exclusive features. Some of these are related to compliance, like guaranteed message delivery (ALTP) and encrypted, time-stamped log storage (LogStore). Others are cloud-related, like the Azure Sentinel destination, or Google Pub/Sub support. Another notable difference between OSE and PE is support. OSE is supported by the community, while PE users are supported by One Identity. Many users chose PE not because of the exclusive features but for the enterprise-grade support.
No matter which edition of syslog-ng you use, the basic functionality and architecture is the same. They collect log messages, process them (parse, normalize, rewrite, and so on) and filter them. Finally, they either store log messages locally or forward them to a remote location. The architecture is usually multi-tiered: clients (individual hosts) forward logs to relays, which in turn forward logs to another level of relays or a central syslog-ng server. This ensures that logs are immediately sent from the hosts and logs are queued in case the central server is unavailable. The central syslog-ng server often routes log messages to multiple log analysis software, locally and in the cloud as well.
Before you begin
Version 7.0.21 of syslog-ng PE was the first one to support Google Pub/Sub. As syslog-ng features receive performance and bug fixes, even minor feature enhancements after the original feature releases, it is recommended to always use the latest feature release. If you do not have syslog-ng PE yet, download the 30-day free trial from https://www.syslog-ng.com/register/115582/
You need a recent syslog-ng PE version to test Azure Event Hub. As usual, I recommend the latest feature release.
Configuring syslog-ng PE for Azure Event Hub
Sending logs to Azure Event Hub demonstrates the flexibility of the syslog-ng http() destination. There is no dedicated Azure Event Hub destination yet in syslog-ng PE, but you can easily configure the http() destination to send logs there. Many drivers in syslog-ng use the http() destination in the background, for example the elasticsearch-http() destination. They hidesome of the implementation complexity.
The following configuration snippet uses the http() destination to send log messages to the Azure Event Hub. Refer to https://learn.microsoft.com/en-us/rest/api/eventhub/generate-sas-token and to our webinar (https://www.syslog-ng.com/webcast-ondemand/feeding-events-to-azure-event-hubs-google-pub-sub/) on how to generate the required shared access signature token. You need to replace fields enclosed in {} with your actual values.
destination d_AzureEventHub {
http(
url("https://{HubNameSpace}.servicebus.windows.net/{HubName}")
method("POST")
headers(
"Authorization: {Shared_Access_Signature_Token}",
"Content-Type: application/atom+xml;type=entry;charset=utf-8"
)
tls( ca-file("{path_to_root_certificate}") )
);
};
Configuring syslog-ng PE for Google Pub/Sub
Sending logs to Google Pub/Sub utilizes the built-in google_pubsub() destination driver. It requires a JSON key for a service account with Pub/Sub Publisher permissions. This destination driver provides At-Least-Once delivery. What it means is that in case of a mid-batch error, the entire batch is resent even if some messages have already been acknowledged.
Replace fields enclosed in {} with your actual values in the configuration below:
destination d_Google_PubSub {
google_pubsub(
credentials("{key.json}")
gcp_auth_header_params(ca_file({path_to_root_certificate}))
project({project})
topic({topic})
);
};
What is next?
If you want to try feeding events with syslog-ng PE to Azure Event Hub or Google Pub/Sub, watch our webcast at https://www.syslog-ng.com/webcast-ondemand/feeding-events-to-azure-event-hubs-google-pub-sub/ which helps you to set up a test environment and demonstrates both cloud destinations in action. It also gives you a good introduction to syslog-ng PE.