Introduction to the Python HTTP header

You can create your own custom headers for the HTTP destination using the Python HTTP header plugin of syslog-ng and Python scripts. The included example configuration just adds a simple counter to the headers but with a bit of coding you can resolve authentication problems or fine tune how data is handled at cloud-based logging and SIEM platforms, like Sumologic.

How it works?

Like all other Python bindings in syslog-ng, there are two parts of the Python HTTP header. One is the syslog-ng configuration. In this case, it is part of the HTTP destination configuration. The other part is the actual Python code, which can be either included in the syslog-ng configuration or stored in an external Python script.

Configuration

The python-http-header configuration is part of the HTTP destination configuration. Here is an example:

destination d_http {
    http(
        python_http_header(
            class("TestCounter")
            options("header", "X-Test-Python-Counter")
            options("counter", 11)
            # this means that syslog-ng will trying to send the http request even when this module fails
            mark-errors-as-critical(no)
        )
        url("http://127.0.0.1:8888")
    );
};

From the listed options, only class() is mandatory. This defines the name of the Python class, which implements the Python HTTP header.

Values defined in options() are passed to the __init__() method of the class. This way you pass initial values to the Python code from the syslog-ng configuration.

The mark-errors-as-critical() defines what happens if the Python code fails for whatever reasons. If set to “yes”, messages are discarded. Use this if you use Python HTTP header for authentication, as after a failed authentication you cannot send messages anyway. Set it to “no” if headers carry only useful but not mandatory information. For example, when you send logs to Sumologic, headers can contain extra information, but they can be useful and can reach the destination even without the extra fields. In either case, an error message is generated where you can see what happened.

Python code

You can store the Python code either in the syslog-ng configuration or externally. There are many advantages of storing code externally but it also adds complexity. For shorter, easy to understand code, including the code in the configuration, it is easier. In this case, it needs to be enclosed in a python {} block:

python {
from syslogng import Logger
logger = Logger()
class TestCounter():
    def __init__(self, options):
        self.header = options["header"]
        self.counter = int(options["counter"])
        logger.debug(f"TestCounter class instantiated; options={options}")
    def get_headers(self, body, headers):
        logger.debug(f"get_headers() called, received body={body}, headers={headers}")
       
        response = ["{}: {}".format(self.header, self.counter)]
        self.counter += 1
        return response
    def __del__(self):
        logger.debug("Deleting TestCounter class instance")
};

You can see here the same class name as defined in the configuration part. Below that there are three methods. From those only get_headers() is mandatory. It receives both the existing headers and the body from syslog-ng. The latter is useful, when you need to create a header based on the content of the body. The get_headers method returns one or more headers to syslog-ng. The above code implements a very simple counter where the value of the counter is incremented each time a new HTTP request is sent.

The optional __init__() method initializes python-http-headers based on options received from the syslog-ng configuration. In the above code the __init__() method configures the name of the header and sets the initial value of the counter.

The optional __del__() method is run when syslog-ng is stopped or reloaded. Here it only creates a debug-level message indicating that it was executed.

Note: the above code only works when you have a single HTTP worker configured in syslog-ng. If you have multiple workers, the Python code will run in concurrent mode. In this case you need to set up locks around counter manipulation.

Testing

All you need for testing:

syslog-ng 3.26 (or later) with Python support installed

• two open terminal windows

• netcat

Preparations

If your operating system does not have syslog-ng 3.26 (or later) available, check our third party repositories page. On most Linux distributions, features requiring extra dependencies are available in sub packages. For Python support, it means that you also need to install a package called syslog-ng-python, syslog-ng-mod-python, or similar. In case of openSUSE, the netcat (nc) command is installed by default and is part of the netcat-openbsd package. The example configuration is based on the configuration snippets from above. We also add a network source and connect the two using a log statement. On most Linux distributions, you can create configuration snippets under /etc/syslog-ng/conf.d with a .conf extension. Otherwise append the following to syslog-ng.conf:

python {
from syslogng import Logger
logger = Logger()
class TestCounter():
    def __init__(self, options):
        self.header = options["header"]
        self.counter = int(options["counter"])
        logger.debug(f"TestCounter class instantiated; options={options}")
    def get_headers(self, body, headers):
        logger.debug(f"get_headers() called, received body={body}, headers={headers}")
       
        response = ["{}: {}".format(self.header, self.counter)]
        self.counter += 1
        return response
    def __del__(self):
        logger.debug("Deleting TestCounter class instance")
};
source s_network {
  network(port(5555));
};
destination d_http {
    http(
        python_http_header(
            class("TestCounter")
            options("header", "X-Test-Python-Counter")
            options("counter", 11)
            # this means that syslog-ng will trying to send the http request even when this module fails
            mark-errors-as-critical(no)
        )
        url("http://127.0.0.1:8888")
    );
};
log {
    source(s_network);
    destination(d_http);
    flags(flow-control);
};

Once you saved your configuration and reloaded syslog-ng, you are ready for testing.

Testing

In this simplified example we only receive the first HTTP message from syslog-ng. The reason is simple: netcat is easily available, but it cannot properly respond to HTTP requests, it just logs what it receives. Start it listening on port 8888:

nc -l 8888

Now send a syslog message to port 5555 of syslog-ng using logger:

logger --rfc3164 -T -n 127.0.0.1 -P 5555 this is a test massage1

Once you hit enter, you should see a similar output in the terminal window where netcat is running:

POST / HTTP/1.1
Host: 127.0.0.1:8888
User-Agent: syslog-ng 3.26.1/libcurl 7.60.0
Accept: */*
X-Syslog-Host: localhost
X-Syslog-Program: root
X-Syslog-Facility: user
X-Syslog-Level: notice
X-Test-Python-Counter: 11
Content-Length: 23
Content-Type: application/x-www-form-urlencoded

this is a test massage1

While the above example is not even a fully working demo (you do not see the counter to increment) it is more than enough to see that adding headers from Python code works. The extra header is there with the name and initial value as set in the configuration.

What is next?

Now, that you have seen Python HTTP header in action, it’s time to develop your own code. The above example is a good starting point. It has all the possible components and it also has debug logging enabled. If you start syslog-ng in the foreground (syslog-ng -Fvde), you can see on screen what initial options the Python code received and the headers and body of the HTTP request sent by syslog-ng.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.