Summary
I set up a small virtulization server by installing FreeBSD using my desktop computer with Dnsmasq, an old Netgear router, mfsBSD and a ThinkCentre M710q booting over PXE. It was not as hard as I thought but still pretty involved compared to dd-ing an ISO file on a usb stick. This post describes the general process I followed to get the thingie to boot a fresh FreeBSD installation.
Setting the PXE server
The domestic network environment in which the PXE server and the target PXE booting client looks like this:
ISP router =>
consumer-grade Netgear router (10.0.0.1) =>
- PXE server (10.0.0.2)
- Soon-to-be-FreeBSD server (10.0.0.136)
Said PXE server is actually my desktop computer on Debian. I followed part of this guide (in French) to install and configure the different components. Using this method, the PXE server is used as both a DCHP server and a PXE server for the client. I didn’t want to disable DHCP from the ISP router, so I disabled it on the intermediate router to be sure the client will get its IP from the PXE server. The client also obviously needs to be configured to boot on PXE in the BIOS.
The configuration process I followed is almost the same as the one described in the guide. I’ll provide the Dnsmasq configuration file and the content of the menu a bit later.
Installing Dnsmasq and the bootloaders. Dnsmasq And The Bootloaders sounds like the name of indie rock band.
apt install dnsmasq pxelinux syslinux-common
Preparing the TFTP root folder. Dnsmasq provides this feature on top of being a DNS and DHCP server, which I didn’t understand at first. The root folder will contains the ISO files to boot from, the bootloaders and a default custom menu file to select the images from the client:
mkdir -p /opt/tftpboot/pxelinux.cfg
mkdir -p /opt/tftpboot/_iso
cd /opt/tftpboot
touch pxelinux.cfg/default
cp /usr/lib/PXELINUX/pxelinux.0 .
cp /usr/lib/syslinux/memdisk .
cp /usr/lib/syslinux/modules/bios/* .
cd /opt/tftpboot/_iso/
wget <any_required_iso> -O <name.iso>
...
vim.tiny ../pxelinux.cfg/default
Backing up the default Dnsmasq configuration file just in case, editing the new configuration and restarting the service:
mv /etc/dnsmasq.conf /etc/dnsmasq.conf.old
vim.tiny /etc/dnsmasq.conf
systemctl restart dnsmasq.service
Things got a bit south from this point. First, this ThinkCentre machine is a UEFI device. DnsMasq proxyDHCP does not work on UEFI environments.
Then, monitoring the server using journalctl -f --unit dnsmasq, I could see some movement but the client failed to get anything from the server.
sept. 21 10:41:19 gargantua dnsmasq-dhcp[64549]: 2066689538 sent size: 35 option: 43 vendor-encap ...
sept. 21 10:41:20 gargantua dnsmasq-dhcp[64549]: 106422341 available DHCP subnet: 10.0.0.0/255.255.255.0
sept. 21 10:41:20 gargantua dnsmasq-dhcp[64549]: 106422341 vendor class: PXEClient:Arch:00007:UNDI:003016
Looking for PXEClient:Arch:00007:UNDI:003016 online revealed that some more options for DHCP discovery were required in Dnsmasq configuration file. See the previous link to the FOG wiki for more information.
Finally, the mysterious tftp time out error I got on the client was exactly what it described: I forgot to put the enable-tftp instruction back in the file.
Below are the pxelinux.cfg/default and /etc/dnsmasq.conf I ended up using to successfuly get the ISO to load on the client. In this case it’s a pot-pourri of things I found online and tried until it worked, because this ghetto install is (hopefully) a one-shot. For syntax reference, start with the dnsmasq manual and syslinux wiki.
DEFAULT menu.c32
MENU TITLE PXE Install
LABEL reboot
MENU DEFAULT
MENU LABEL Reboot
COM32 reboot.c32
label mfsbsd
menu label ^mfsbsd 14
kernel memdisk
initrd _iso/mfsbsd_14.iso
APPEND iso raw
enable-tftp
tftp-root=/opt/tftpboot
port=0
log-dhcp
dhcp-no-override
dhcp-match=set:efibc,60,PXEClient:Arch:00007
dhcp-boot=tag:efibc,ipxe.efi,10.0.0.2,10.0.0.2
dhcp-range=10.0.0.0,10.0.0.255
dhcp-vendorclass=BIOS,PXEClient:Arch:00000
dhcp-vendorclass=UEFI32,PXEClient:Arch:00006
dhcp-vendorclass=UEFI,PXEClient:Arch:00007
dhcp-vendorclass=UEFI64,PXEClient:Arch:00009
dhcp-boot=net:UEFI32,i386-efi/ipxe.efi,,10.0.0.2
dhcp-boot=net:UEFI,ipxe.efi,,10.0.0.2
dhcp-boot=net:UEFI64,ipxe.efi,,10.0.0.2
pxe-prompt="Booting FOG Client", 1
pxe-service=X86PC, "PXELinux", pxelinux
Installing FreeBSD from mfsBSD
This configuration allowed me to get the FreeBSD 14.1 iso to load on the client. Installing it was cut short by an error interupting the boot process on a mountroot prompt:
cd9660 :/dev/iso9660 error :/dev/iso9660/FREEBSD_INSTALL failed with error 19`
Troubleshooting that, I realized the consensus was that booting a stock FreeBSD over PXE was still not really a thing in current year but that people had much more success using mfsBSD and running bsdinstall from there. So here it is:
cd /opt/tftpboot/_iso
wget https://mfsbsd.vx.sk/files/iso/14/amd64/mfsbsd-14.1-RELEASE-amd64.iso -O mfsbsd_14.iso
And quite accurately, the initial boot process succeeded on a nice prompt, and I was able to log in over SSH with the credentials root/mfsroot and start the installation process with bsdinstall.
The main hickup experienced during the installation was related to DHCP. The Internets were not reachable because the machine got its IP from the PXE server.
root@mfsbsd:~ # service dhclient restart em0
Stopping dhclient.
Waiting for PIDS: 2643.
Starting dhclient.
DHCPREQUEST on em0 to 255.255.255.255 port 67
DHCPACK from 10.0.0.2
bound to 10.0.0.136 -- renewal in 1800 seconds.
I didn’t want this interface to contact the PXE server to get its DHCP lease anymore. A quick look at /var/db/dhclient.leases.em0 showed that it still did, obviously. I first tried to configure /etc/dhclient.conf to reject the PXE host’s DHCPACK offer, but it didn’t work. I ended up reactivating DHCP on the Netgear router and restarting dhclient again.
Mostly smooth sailing from there. Ensure the local FreeBSD mirror you choose is still up before starting the install process, or that you know what to do if it aborts after it partionned the disk and mounted the volume. Also, be sure that you add a local user during the installation process: The default SSH configuration for FreeBSD forbids root logins.