Fail2ban and Dockers's journald logging driver

Summary

The following post describes the procedure followed to configure Docker (rootless) to log the activity of its containers using the journald driver and adapt Fail2ban’s settings to read from it. I mostly had to fish for bits of information in issues or forum posts, so this article may help you achieve the same thing a bit faster.

Why would you want to do that

  • I don’t want to run logrotate alongside Nginx in the same container, having to mess with cron and permission issues on top of that.
  • I don’t want to do bind mounts to get access to container logs.
  • journactl works well enough, its filtering abilities are handy. I made peace with Systemd. Kind of.
  • Docker’s default logging driver, json-file, doesn’t rotate logs at all, which will eventually cause problems even for a small server.

Configure Docker

The first steps to achieve this is to change the logging driver for dockerd. In /etc/docker/daemon.json, on in my case .config/docker/daemon.json since I’m running Docker rootless, change the settings so that dockerd use the desired driver :

{
  "log-driver": "journald"
}

Then restart the daemon with systemctl --user restart docker. For the containers to use the new logging driver, they also need to be restarted. I recreated everything with docker-compose up -d --build --force but that may be overkill. On the systemd side of things, you should now be able to see the container logs with journalctl CONTAINER_NAME=<name>.

Configure the Nginx container

A little more work for Nginx was required, since I used different log paths for earch virtual host.

The default log paths in /etc/nginx/nginx.conf should be left out. The symbolic links to /var/log/nginx/access.log and /var/log/nginx/access.log should not be removed. They redirect the logs to the standard output, which makes them readable from journalctl. Log formatting can of course be modified but be careful to adapt the matching regex in Fail2ban’s filters.

ls -l /var/log/nginx/
total 0
lrwxrwxrwx 1 root root 11 Jul  4 19:25 access.log -> /dev/stdout
lrwxrwxrwx 1 root root 11 Jul  4 19:25 error.log -> /dev/stderr

Configure Fail2ban

Fail2ban must be configured to use the systemd backend. Prior to any modification, the Python bindings to Systemd must be installed for Fail2ban to be able to query the API parse the logs with this backend. On a $current_year Debian, this can be done with apt install python3-systemd. Once done, the jail.local file can be updated accordingly.

[DEFAULT]
backend = systemd[journalflags=1]
...
action_mwm = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
             %(mta)s-whois-matches[name=%(__name__)s, dest="%(destemail)s", chain="%(chain)s", sender="%(sender)s", sendername="%(sendername)s"]
action = %(action_mwm)s
...
[nginx-noscript]
enabled = true
filter = nginx-noscript[journalmatch='CONTAINER_NAME=nginx'] 
...
[gitea]
enabled = true
filter = gitea[journalmatch='CONTAINER_NAME=gitea']

From the example above :

  • The backend option is updated to systemd[journalflags=1]. Without the flag, Fail2ban won’t be able to see the logs from the containers and will only parse system logs.
  • If mail notification is configured, be sure to use the sendmail-whois-matches action rather than sendmail-whois-lines, which doesn’t support parsing logs this way and expect a logpath pointing to a file.
  • The journalmatch option should be passed to the filter, else It will parse the whole journalctl output rather than the relevant logs from the container/service/unit. It can alternatively be defined in the filter configuration file.

The syntax for testing regexes and filters with fail2ban-regex is slightly different in that systemd-journal is the argument replacing the log path. For example : fail2ban-regex --journalmatch="CONTAINER_NAME=nginx" systemd-journal[journalflags=1] "nginx-botsearch"

Conclusion

It’s a bit convoluted and not well documented but it does the job. Fail2ban on my server is configured this way, with specific jails for each exposed services running inside rootless containers, managed by a dedicated user with limited privileges. Those measures might be sufficient for a single self-hosted potato without critical data to protect.

References