The latest version of sudo, version 1.9.4 includes support for JSON formatted logging. Compared to traditional sudo logs, it has the advantage of containing more information in a structured way. While traditional sudo logs are also parsed automatically by syslog-ng, it is worth taking a look at the new JSON formatted logs.
From this blog, you can learn how the new logs look like and also a configuration working with these logs. Instead of just posting a complex configuration, I try to show you how my configuration was built. Creating a new configuration in smaller iterations makes the resulting configurations easier to debug.
Before you begin
JSON formatted logging first appeared in sudo 1.9.4, released this week. It is not yet available in Linux distributions. You can install it only from source or using binaries on the sudo website. On the syslog-ng side I used version 3.30, but any recent version should be OK, such as 3.20+.
Configuring sudo
You can enable JSON logging in sudo with a single line in the sudoers file. Start visudo and add this line:
Defaults log_format = json
The next time you start sudo you should already see JSON formatted log messages. You will see a line similar to this in /var/log/messages or /var/log/secure (CentOS):
Nov 30 10:00:37 centos8splunk.localdomain sudo[14170]: @cee:{"accept":{"server_time":{"seconds":1606726837,"nanoseconds":204921788,"iso8601":"20201130090037Z","localtime":"Nov 30 09:00:37"},"submit_time":{"seconds":1606726835,"nanoseconds":164020105,"iso8601":"20201130090035Z","localtime":"Nov 30 09:00:35"},"submituser":"czanik","command":"/usr/bin/ls","runuser":"root","runcwd":"/home/czanik","ttyname":"/dev/pts/0","submithost":"centos8splunk.localdomain","submitcwd":"/home/czanik","runuid":0,"columns":118,"lines":53,"runargv":["ls","/root/"]}}
Configuring syslog-ng
As you can see from the log sample, the header of the log is exactly the same as with the traditional sudo logs: it contains a date, a host name and a program name with a process ID. The difference is in the message part. It starts with a marker “@cee:” and then continues with a JSON formatted log message.
Saving sudo logs separately
As a first step, let us create a configuration, which collects all sudo logs into a separate log file. The same configuration would also work with traditional sudo logs, as filtering is based on the program name included in the message header. Note that the source name might be different on your system: check the name of the local log source in syslog-ng.conf. In Fedora and CentOS syslog-ng packages, “s_sys” is used.
filter f_sudo {
program(sudo);
};
destination d_sudo {
file("/var/log/sudo");
};
log {
source(s_sys);
filter(f_sudo);
destination(d_sudo);
};
Append this to syslog-ng.conf or save it under a separate configuration file with a .conf extension under the /etc/syslog-ng/conf.d/ directory, if your Linux distribution of choice supports this setup. Do not forget to restart or reload syslog-ng for the configuration to take effect.
From now on, sudo logs are also filed into a separate file. If you run sudo now, you should see a similar log message in /var/log/sudo.
Parsing JSON logs
The next step is to parse JSON logs. If we just add a parser then we do not see the results of the parsing. So we also add a template to the file destination to use the results of message parsing.
filter f_sudo {
program(sudo);
};
destination d_sudo {
file("/var/log/sudo" template("$(format-json --scope rfc5424 --scope nv-pairs)\n\n"));
};
parser p_sudo {
json-parser(
marker("@cee:")
prefix("sudo.")
);
};
log {
source(s_sys);
filter(f_sudo);
parser(p_sudo);
destination(d_sudo);
};
For the JSON parser we use two parameters. When the marker() is set, the JSON parser of syslog-ng tries to parse only messages starting with the marker text. In our case it is “@cee:”. The prefix() tells the parser what prefix to use in the name of the resulting name-value pairs. In our case it is “sudo”. This way, names coming from parsing do not clash with internal name-value pairs of syslog-ng. Note the dot at the end of name. The syslog-ng JSON template function creates nested JSON when it encounters a dot in the name of a name-value pair.
The template in the file destination adds any syslog related name-value pairs and also name-value pairs parsed from the message in JSON format. The two extra line feeds make the log file more human-readable.
Here is an example log of a rejected session:
{"sudo":{"reject":{"ttyname":"/dev/pts/0","submituser":"czanik","submithost":"centos8splunk.localdomain","submitcwd":"/home/czanik","submit_time":{"seconds":"1606723559","nanoseconds":"260437687","localtime":"Nov 30 08:05:59","iso8601":"20201130080559Z"},"server_time":{"seconds":"1606723566","nanoseconds":"751959887","localtime":"Nov 30 08:06:06","iso8601":"20201130080606Z"},"runuser":"root","runuid":"0","runenv[9]":"SHELL=/bin/bash","runenv[8]":"HOME=/root","runenv[7]":"USER=root","runenv[6]":"LOGNAME=root","runenv[5]":"MAIL=/var/mail/root","runenv[4]":"PATH=/home/czanik/.local/bin:/home/czanik/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin","runenv[3]":"TERM=xterm-256color","runenv[2]":"HOSTNAME=centos8splunk.localdomain","runenv[1]":"LANG=en_US.UTF-8","runenv[13]":"SUDO_GID=1000","runenv[12]":"SUDO_UID=1000","runenv[11]":"SUDO_USER=czanik","runenv[10]":"SUDO_COMMAND=/usr/bin/ls /root/","runenv[0]":"LS_COLORS=rs=0:di=38;5;33:ln=38;5;51:mh=00:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=01;05;37;41:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;40:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.zst=38;5;9:*.tzst=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.wim=38;5;9:*.swm=38;5;9:*.dwm=38;5;9:*.esd=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.mjpg=38;5;13:*.mjpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.m4a=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.oga=38;5;45:*.opus=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:","runcwd":"/home/czanik","runargv[1]":"/root/","runargv[0]":"ls","reason":"3 incorrect password attempts","lines":"53","command":"/usr/bin/ls","columns":"118"}},"SOURCE":"s_sys","PROGRAM":"sudo","PRIORITY":"alert","PID":"7292","MESSAGE":"@cee:{\"reject\":{\"reason\":\"3 incorrect password attempts\",\"server_time\":{\"seconds\":1606723566,\"nanoseconds\":751959887,\"iso8601\":\"20201130080606Z\",\"localtime\":\"Nov 30 08:06:06\"},\"submit_time\":{\"seconds\":1606723559,\"nanoseconds\":260437687,\"iso8601\":\"20201130080559Z\",\"localtime\":\"Nov 30 08:05:59\"},\"submituser\":\"czanik\",\"command\":\"/usr/bin/ls\",\"runuser\":\"root\",\"runcwd\":\"/home/czanik\",\"ttyname\":\"/dev/pts/0\",\"submithost\":\"centos8splunk.localdomain\",\"submitcwd\":\"/home/czanik\",\"runuid\":0,\"columns\":118,\"lines\":53,\"runargv\":[\"ls\",\"/root/\"],\"runenv\":[\"LS_COLORS=rs=0:di=38;5;33:ln=38;5;51:mh=00:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=01;05;37;41:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;40:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.zst=38;5;9:*.tzst=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.wim=38;5;9:*.swm=38;5;9:*.dwm=38;5;9:*.esd=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.mjpg=38;5;13:*.mjpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.m4a=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.oga=38;5;45:*.opus=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:\",\"LANG=en_US.UTF-8\",\"HOSTNAME=centos8splunk.localdomain\",\"TERM=xterm-256color\",\"PATH=/home/czanik/.local/bin:/home/czanik/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin\",\"MAIL=/var/mail/root\",\"LOGNAME=root\",\"USER=root\",\"HOME=/root\",\"SHELL=/bin/bash\",\"SUDO_COMMAND=/usr/bin/ls /root/\",\"SUDO_USER=czanik\",\"SUDO_UID=1000\",\"SUDO_GID=1000\"]}}","HOST_FROM":"centos8splunk","HOST":"centos8splunk.localdomain","FACILITY":"authpriv","DATE":"Nov 30 09:06:06"}
The blog engine prints it on a single line, but otherwise it would fill almost two pages as it contains tons of information about the rejected session.
Alerting (filtering) on extracted fields
As a last step, we add simple alerting to our configuration. From the technical point of view, alerting in syslog-ng practically means filtering. In this example, I add a file destination. In the real world, you would add a Slack or SMTP destination for alerts. Even with a fancy remote destination, you would most likely start with a file destination for testing your filters.
Here I combined two steps into one: filtering the messages and formatting the output. When I had only the filter sending logs to a file, it worked fine. On the other hand, imagine receiving the two-page-long message on Slack. You would want to receive only the important data and check out the rest when you have time to respond to the alert.
The “if” statement is checking if the failed session was initiated with my user name, and writes the user name and the reason of the failure in the file.
filter f_sudo {
program(sudo);
};
destination d_sudo {
file("/var/log/sudo" template("$(format-json --scope rfc5424 --scope nv-pairs)\n\n"));
};
parser p_sudo {
json-parser(
marker("@cee:")
prefix("sudo.")
);
};
log {
source(s_sys);
filter(f_sudo);
parser(p_sudo);
if (match("czanik" value("sudo.reject.submituser"))) {
destination { file("/var/log/sudo_danger" template("$(format-welf --key sudo.reject.submituser --key sudo.reject.reason)\n")); };
# ToDo: change to a slack or smtp destination
};
destination(d_sudo);
};
When you enter a wrong password, you can see a similar log in the /var/log/sudo_danger file:
sudo.reject.reason="3 incorrect password attempts" sudo.reject.submituser=czanik
What is next?
As mentioned and also commented in the example configuration, for a real world situation you will want to add an SMTP, Slack or similar destination for alerts. Of course, for testing the file destination is good enough and even encouraged. You will most likely want to filter on a different user name or a different field, like get alerted any time a user runs a shell (match on “bash”).
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @Pczanik.