In version 4 of syslog-ng, the role of Python became even more important. Previously, all parts of syslog-ng could be extended using Python code, but no actual Python code was provided with syslog-ng. Version 4.0 added a Kubernetes module implemented in Python, while version 4.2 added support for Hypr. But how can we ensure that all Python dependencies are met?
Current situation
Of course, Python modules add their own long list of dependencies. There are three ways how these dependencies can be installed:
-
On the system using the package management of the OS. This is the preferred way of many organizations, but unfortunately not all dependencies are available for all operating systems as ready-to-install packages.
-
On the system using Python tools. This is not recommended, unless you really know what you are doing, as mixing OS-maintained packages with manually installed components can easily lead to chaos.
-
Using a syslog-ng provided script (syslog-ng-update-virtualenv) to create a syslog-ng specific venv (virtual environment).
From a packaging point of view, right now there are three different approaches, all with their own advantages and disadvantages.
-
Packages made by the syslog-ng developers run the syslog-ng-update-virtualenv script automatically during package installation for the syslog-ng-mod-python package. The main advantages of this approach are that there are no manual steps, the installation ensures that all dependencies are available immediately and the versions are tested by syslog-ng developers. However, some sites do not have any Internet access, or the connection they have is very slow. Also, downloading non-OS packages is often not welcome or prohibited completely in the environment.
-
This is why I use OS-provided Python dependencies in my RPM packages, where all necessary dependencies are available. This means no extra downloads, only OS-provided packages. This includes all my official and unofficial openSUSE and SLES packages, along with Fedora and unofficial RHEL 9 packages.
-
Where the OS packages for dependencies are unavailable, my packages do not run syslog-ng-update-virtualenv automatically. This is more compliant to stricter regulations, but also means that syslog-ng does not start, unless someone runs the script manually. Upgrading to a new syslog-ng version can also lead to syslog-ng not starting up. This includes FreeBSD ports and unofficial RHEL 7 and 8 packages.
My packaging plans
Right now, if you install syslog-ng Python support (which means installing the syslog-ng-python RPM package, or compiling FreeBSD ports with Python support) then Python-based syslog-ng modules are also installed. While the actual Python code in syslog-ng is small, they pull in well over 50 MB of dependencies, many times the size of a syslog-ng installation. Most syslog-ng users do not run Kubernetes, especially FreeBSD users and people running older Linux distro versions. For them, the extra installation time and disk space could be especially painful. Also, those who forget to run the script may get some really ugly error messages:
root@fb132:/var/db # syslog-ng -s
[2023-05-12T15:24:20.294277] python: private virtualenv is not initialized, use the `syslog-ng-update-virtualenv' script to initialize it or make sure all required Python dependencies are available in the system Python installation; path='/var/db/python-venv'
[2023-05-12T15:24:20.342641] Error evaluating global Python block; exception='ModuleNotFoundError: No module named \'requests\''
Traceback (most recent call last):
File "/usr/local/etc/syslog-ng.conf{python-global-code:25}", line 3, in <module>
import syslogng.modules.hypr
File "/usr/local/lib/syslog-ng/python/syslogng/modules/hypr/__init__.py", line 48, in <module>
import requests
ModuleNotFoundError: No module named 'requests'
Error parsing python, Error processing global python block in /usr/local/lib/syslog-ng/python/syslogng/modules/hypr/scl/hypr.conf:25:1-25:7:
20 # OpenSSL libraries as published by the OpenSSL project. See the file
21 # COPYING for details.
22 #
23 #############################################################################
24
25----> python {
25----> ^^^^^^
26
27 import syslogng.modules.hypr
28
29 syslogng.modules.hypr.register_hypr_config_generator()
30
Included from /usr/local/share/syslog-ng/include/scl/python/python-modules.conf:25:1-25:1:
Included from /usr/local/etc/scl.conf:31:1-31:1:
26 #
27
28 @module appmodel
29
30 @include 'scl/*/*.conf'
31---->
31----> ^
32 @define java-module-dir "`module-install-dir`/java-modules"
Included from /usr/local/etc/syslog-ng.conf:3:1-3:1:
1 @version:4.2
2 @include "scl.conf"
3----->
3-----> ^
4 #
5 # This sample configuration file is essentially equilivent to the stock
6 # FreeBSD /etc/syslog.conf file.
7 #
8
Because of these, I plan to split the syslog-ng Python module from the actual Python-based drivers in my packages. Or, in case of FreeBSD, make the installation of Python-based drivers a separate option.
This has two advantages:
-
A considerably smaller disk space requirement.
-
A much smaller chance to see error messages like the one above.
Those who need Kubernetes or Hypr support can easily install the separate package, which will also automatically install OS-provided Python dependencies where available.
Feedback
I do syslog-ng packaging based on my own experiences and the feedback reaching me. As such, I opened a discussion on GitHub: https://github.com/syslog-ng/syslog-ng/discussions/4481 You can share your own ideas about how to package syslog-ng Python support for openSUSE / SLES, Fedora / RHEL and FreeBSD.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.