On FreeBSD Jails, coming from Docker

Thoughts and opinions on what I learned, often the hard way, while configuring a FreeBSD server behind a CG-NAT and migrating a small Docker infrastructure to a Jail-based one: Networking woos, trade-offs and web services gotchas.

Introduction

Around two years ago, I started working on setting up a FreeBSD server on a tiny machine for various purposes: Web services, file storage, network access, virtual environments for security-related endeavors, and just having a general learning ground.

The initial setup was promptly wiped up by the combined power of a borked update, disk failure and shitty backups. After half a year learning new skills far away from a computer screen and a lot of hair-pulling once I got back in front of it, the thingie is up and running once again.

I had prior, distant experiences using FreeBSD on a desktop machine, but not on a server. It’s not as smooth of a transition as you could think. From a bird view everything looks alike, especially in user-space. But the level of familiarity you expect from popping in a Linux box is challenged as soon as the installation ends.

In this article, I’ll specifically focus on Jails, which was one of the main selling point for trying FreeBSD, alongside pf and ZFS.

On jails

Docker and jails: Not much in common

“No shit Sherlock” I hear you say.

The idea of switching on a FreeBSD infrastructure with jails might have stemmed from an afternoon trying LXC and giving up after realizing I was, yet again, struggling with software layers clashing with each others because of poor integration rather than my own misunderstanding of a technology new to me, which is a recurring experience in the Ganoo+Linux world. I don’t remember if it was SystemD, Apparmor, both or something else, but I was fed up spending so much time debugging weird system-specific issues while getting no actual work done.

Surely switching on a container solution integrated into the base operating system since the year 2000 should allows me to actually focus on getting things to work inside those containers? You bet it did.

Jails are rightly known to be a simple, robust, and pretty well integrated system, and that’s my experience so far with the small network of jails I set up. There’s no registry, no “images” to build, no entrypoints. In practice, I’m basically manipulating light VM-like objects isolated from the host, managed like a host. As far as I know, I don’t have to worry much about security if I don’t modify the default securelevel settings.

There’s obvious downsides. You’re basically trading off a lot of what makes Docker “just work” against more flexibility and way less magic. The main pain point for me was learning to set up dedicated networks for the jails, as I was so used to Docker creating firewall rules and interfaces without my intervention. This means you’ll have to dive into pf and ifconfig manuals.

Other than that, it’s not so bad. I’m fine with ZFS cloning to create new jails and PyInfra to automate the repetitive tasks, but Jail management tools exist: IOcage, Bastille, etc. Shell scripts in /usr/local/libexec automate network interfaces (currently epairs) set up and tear down. The port collection is extensive, and most services I used on my previous Docker infra are just a pkg install <thing> away. Some services don’t support FreeBSD well (Invidious) or expect you to install them with Docker or Podman (CI/CD things like Woodpecker). Though I didn’t dig into this subject, I’m aware there are efforts to bring OCI containers into FreeBSD with Podman.

RAM usage. Docker rootless network stack VS Pleroma

Jail networking: What should I use?

I initially set up the networking for my jails as shared IP with the host interface. It was confusing and I experienced all kind of routing issues. Be aware that the handbook is not a manual, and it applies to setting up jail networks as well.

Probing FreeBSD forums and the World Wide Web for blogposts on the matter, I came to the conclusion that using VNET and epairs was actually the easiest choice for me to meet a good level of isolation between the different jails subnets without having to write weird rules in the firewall.

The network for jails on this server is currently configured like this:

A bridge0 interface is first added (ifconfig bridge create bridge0)

Jails are configured from a common template located at /etc/jail.conf.d/base.conf, which is included in the jail specific file using .include "/etc/jail.conf.d/base.conf";.

# base configuration for jail

# startup/logging
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_console_${name}.log";

# network interfaces
vnet;
vnet.interface = "epair_${short}b";

# hooks
exec.prestart  = "/usr/local/libexec/jail-epair.sh up ${name} ${short} ${bridge}";
exec.poststart = "/usr/local/libexec/jail-net.sh ${name} ${short} ${ip4} ${gw4}";
exec.poststop  = "/usr/local/libexec/jail-epair.sh down ${name} ${short}";

# permissions
mount.devfs;
allow.raw_sockets;
exec.clean;

path = "/usr/local/jails/${name}";

The scripts called by the hooks handle epairs creation and tear down:

jail-net.sh

#!/bin/sh
set -eu

name="${1:?jail name}"
short="${2:?short}"
ip4="${3:?ip/cidr}"
gw4="${4:?gateway}"

jexec "$name" ifconfig "epair_${short}b" inet "$ip4" up

# Make default route idempotent: change if already present
jexec "$name" sh -eu -c '
gw="$1"
if route -n get default >/dev/null 2>&1; then
  route -q delete default >/dev/null 2>&1 || true
fi
route -q add default "$gw"
' sh "$gw4"

jail-epair.sh

#!/bin/sh
set -eu

action="${1:-}"
name="${2:-}"
short="${3:-}"
bridge="${4:-bridge0}"

[ -n "$name" ] || { echo "missing jail name" >&2; exit 2; }
[ -n "$short" ] || { echo "missing short name" >&2; exit 2; }

case "$action" in
  up)
    # If it already exists, do nothing
    if ifconfig "epair_${short}a" >/dev/null 2>&1; then
      ifconfig "epair_${short}a" up
      ifconfig "$bridge" addm "epair_${short}a" || true
      exit 0
    fi

    base="$(ifconfig epair create | sed 's/[ab]$//')"

    # Rename both ends deterministically
    ifconfig "${base}a" name "epair_${short}a"
    ifconfig "${base}b" name "epair_${short}b"

    # Host-side plumbing
    ifconfig "epair_${short}a" up
    ifconfig "$bridge" addm "epair_${short}a"
    ;;

  down)
    # Destroying the a-side destroys the whole pair
    ifconfig "epair_${short}a" destroy >/dev/null 2>&1 || true
    ;;

  *)
    echo "usage: $0 up <jailname> [bridge] | down <jailname>" >&2
    exit 2
    ;;
esac

This way, the process is automated and I can clearly identify each epairs in ifconfig’s output.

I don’t know if there is a better way to do it, but so far it’s as been working great. If it feels cumbersome and frail to you, maybe it’s a good time considering what fully-fledged jail managers can do for you. The default tooling is fine for my use case.

The blog posts in the references section, particularly the one from Klara Systems, should be of great help if you’re still trying to better understand jail, isolation and networking and chose an appropriate solution for your goals.

Thick or thin jails?

I initially settled on using thin jails without even noticing I did by leveraging ZFS snapshot and cloning features.

After setting up a base jail at zroot/jails/basejail-15.0 and making a snapshot of it as described in the handbook, creating a new jail is just a matter of creating the right configuration file in /etc/jail.d/<jailname.conf>, zfs clone it in the target location and starting it:

Create a dataset from the “template” jail:

zfs create zroot/jails/templates/freebsd14.1
tar xvf media/14.1-RELEASE-base.txz -C templates/freebsd14.1/ --unlink
cp /etc/resolv.conf templates/freebsd14.1/
cp /etc/localtime templates/freebsd14.1/

Create a snapshot of the jail: zfs snapshot zroot/jails/templates/freebsd14.1@today

Clone the new jail from the snapshot: zfs clone zroot/jails/templates/freebsd14.1@today zroot/jails/containers/postgresql

Then proceed as usual (/etc/jail.d/tata.conf, etc.)

By doing that you’re giving up some security and robustness for time and convenience. It also implies some basic knowledge about ZFS, which is generally advised to manage a FreeBSD while being a friction area for a Linux user who only ever used EXT3 and EXT4.

Nothing stops you to use both kind of jails depending on your need: Quickly deploying thin jail for testing purpose and setting up thick one for more persistent needs seems to be a common pattern.

Troubleshooting network issues in Jail

Setting the networks for your jails yourself probably means you’ll have to troubleshoot connectivity issues at some point, especially when doing NAT and redirection. I had to do it, a lot. Here are a few helpful commands to locate the issue:

Checking for a default route: netstat -rn
Routing table for ipv6: netstat -rn -f inet6
More information on the default route: route -n get default
Checking connectivity (here with a local DNS server): nc -v 10.10.0.1 53
Checking connectivity (this time with a bare IP): nc -vz 1.1.1.1 853
Query the DNS with the specified server: drill @10.10.0.1 freebsd.org
Print the content of the ARP table: arp -an
Print packets route to public DNS: traceroute -n 1.1.1.1
Check interfaces properties: ifconfig -v | egrep -A6 'bridge|epair'

Also, don’t forget about tcpdump and nc. The later is generally an instant way to know if a service is reachable or not.

Jails don’t start at boot

Even when jail_enable=YES is set in /etc/rc.conf.

Turns out it’s a bug that has not been fixed yet (as of march 2026) where the jails to start need to be specified in jail_list variable.

This is also what define in which order the jail start as far as I know. Useful when some jails depends on core services (database, DNS, etc.).

Off-Topic: First impressions about PyInfra

This is tangential to jails and more about setting up the host server. It doesn’t directly to jails but I still want to fit them in this post.

Pyinfra is perfectly fine if you don’t want neither to reinvent the wheel with shell scripts - like I tried to do - or wrangling YAML files just to update a couple of configuration files or RC scripts.

Coupled with doas set up on the host machine, a few playbooks are all I currently need to update configurations files, validate them and reload the services.

Ansible has its uses, you can find modules for everything under the sun, but my experience with it is that is completely overkill for a small network like that. Even if it was, I’m not programming YAML if I’m not paid to do it, simple as.

We will see how things will turn when I’ll actually want to reap the benefits of idempotency and perform more complex operations but so far it’s perfectly KISS.