Virt
An Opinionated Tech Blog

Home Lab: Giving Old Hardware a Purpose

Table of contents

Due to some rather complicated life circumstances, I moved away from Göttingen. Maybe a midlife crisis, maybe the fear of complacency, it really does not matter.

Anyway, I tackled what I had been neglecting for the last couple of years:

A decent home lab, one where I could tinker, self-host and blog.

Internet connection #

I reside in a rather isolated area now, therefore the Internet connection is limited to 50 Mbit/s. On the bright side, I take the good with the bad, because I really like it here. Place for a garden, I chop wood, I can be loud on Sundays. Perfect.

After researching, the best option available here (for me) was Fonial:

The good with the bad.

Hardware #

I have a rather random collection of hardware from various side projects that I never had the heart to discard. Time to dust it off and make it useful.

The previous tenant left me with a Fritz!Box and a Fritz!Repeater, and I had one boxed up that just collected dust. I am not a particular fan of Fritz!, but I am not against it, it is just too restrictive for my use, but we will work around that later. It does have a nice simple ecosystem. I would go OpenWRT all the way, but I do have them lying around.

My previous setup consisted of a cable modem, a mini PC running OpenBSD. And an access point running OpenWRT. It was awesome! Back then I used only the base install of OpenBSD to handle DHCP, DNS and the Firewall. Trust me, you need to try it. It was the cleanest, most consistent installation I ever did. A good starting point, for those interested, is OpenBSD PF - Building a Router.

Looking through some boxes, I found a ThinkPad T460s with a 1 TB NVME SSD, essentially I just grabbed a free server out of a dusty box. Also, a Redmi cell phone with a cracked screen, looks like we got a free IP camera. I am a cheap bastard, I know. Let me rephrase: I am environmentally conscious, by upcycling old devices, while being economically resourceful.

So here is what we got:

Basic Network Setup #

Nothing too glorious here, reset all router/repeater settings, meshed them together, changed the SSID (to Long Ferryman), added a guest network (LANfermann) and set up a different password. Also changing the DHCP range, from 192.168.44.150 to 192.168.44.254.

It’s pretty easy to set up with Fritz! products, check out the introduction for an overview.

The initial plan, always subject to change, is more or less:

IP Range Function
192.168.44.1 - 192.44.9 Router/Repeater
192.168.44.10 Server
192.168.44.11 - 192.168.44.49 LXC
192.168.44.50 - 192.168.44.99 External devices (TV, IP camera…)
192.168.44.100 - 192.168.44.149 Reserved
192.168.44.150 - 192.168.44.254 Unmanaged DHCP

All done per DHCP on the base station.

Now that the meshing is set up and the IP ranges for DHCP are set in sand, let’s jump right on setting up the server.

The Server #

XUbuntu with ZFS was running on the Server. I used it as a make-shift server a year or so ago, it was pretty much just running Jellyfin, FTP and SMB/CIFS.

I really am dissatisfied with the direction Ubuntu has been going, it’s mostly the god-awful snap package ecosystem, their push for LXD and their fast moving philosophy of breaking my shit. I liked to use it professionally before, since they integrated ZFS on root in the installer and adapted eBPF aggressively. So where to go from here: Debian!

Time for a fresh installation. I need ZFS, I want ZFS, I don’t want to partition the NVME, I want the WHOLE damn device in my pool.

Looking around, I found a microSD with 32 GB of space that worked. The Thinkpad has a card reader, so after throwing Debian 12 (netinstall) onto an old USB-Stick I installed it on the microSD, which was painfully slow and made me a bit worried about the performance.

Of course, after hours of installing Debian 12 an almost minimal installation1, the goddamn Thinkpad didn’t show up the microSD as a boot device. Lucky for me, I found a USB card reader when looking through my stuff before, so problem solved and the performance also increased - I guess the USB 3 slot helped.

So, minimal installation, DHCP, on the Fritz!Box base station stick the IP to 192.168.44.10, name the server plombyr for no particular reason.

ZFS #

To install ZFS, it is necessary to enable the non-free repositories, because Debian maintainers are afraid of the Common Development and Distribution License (CDDL):

Due to potential legal incompatibilities between the CDDL and GPL, despite both being OSI-approved free software licenses which comply with DFSG, ZFS development is not supported by the Linux kernel. ZoL is a project funded by the Lawrence Livermore National Laboratory to develop a native Linux kernel module for its massive storage requirements and super computers.

Mine looks more or less like this now:

$ cat /etc/apt/sources.list
deb http://deb.debian.org/debian bookworm main non-free non-free-firmware contrib
deb http://deb.debian.org/debian bookworm-updates main non-free non-free-firmware contrib
deb http://deb.debian.org/debian-security/ bookworm-security main non-free non-free-firmware contrib

After getting usable packages, let’s install ZFS.

apt update
apt install linux-headers-amd64 zfsutils-linux zfs-dkms zfs-zed

Right, let’s get on with it. Create the pool without a mountpoint:

zpool create -m none trunk /dev/nvme0n1

NOTE: It is RECOMMENDED to use the UUID, in case hard drives are swapped, added, rearranged etc. In my case: It is running on a laptop, I am not changing shit.

Set compression to save disk space, optimize performance by disabling access time logging and allowing extended file attributes:

zfs set compression=lz4 trunk
zfs set atime=off trunk
zfs set xattr=sa trunk

Look up the properties in the man page for zfsprops(7), if those settings elude you. To get a good insight into ZFS I highly recommend the FreeBSD Mastery: ZFS books by Michael W. Lukas and Alan Jude.

The LXC setup #

I am a long time fan of LXC, been using it for more than a decade. Don’t need or want LXD or Docker in my home lab. I also like to use the same IP range and use the Fritz!Box DHCP server for the containers2. To sum it up: I choose LXC over docker with the following network bridge setup to emulate having VPS at home, similar to what Hetzner or Linode offer. I can also just pull up different images for Alpine, openSuse, Ubuntu, etc. if some software needs either bleeding edge packages or outdated ones. I’ll discuss what else I will, or already am, running in containers for my lab in future blog posts.

Let’s install it then:

apt update
apt install lxc bridge-utils

Set up a host device as bridge in Debian first, like so:

$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#allow-hotplug enp0s31f6
#iface enp0s31f6 inet dhcp

# This is an autoconfigured IPv6 interface
iface enp0s31f6 inet6 auto

# LXC network bridge
auto br0
iface br0 inet dhcp
        bridge_ports enp0s31f6
        bridge_fd 0
        bridge_maxwait 0

Remove the default LXC bridge:

$ cat /etc/default/lxc-net
USE_LXC_BRIDGE="false"

# Honor system's dnsmasq configuration
#LXC_DHCP_CONFILE=/etc/dnsmasq.conf

Setup the bridge, and autostart, why not:

$ cat /etc/lxc/default.conf
lxc.net.0.type = veth
lxc.net.0.link = br0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx

lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1

lxc.start.auto = 1

Restart networking:

/etc/init.d/networking restart

Now, let’s combine LXC with ZFS for profit and glory.

Combining ZFS and LXC #

Combining ZFS and LXC gives you superpowers. They are easy to migrate, backup, make snapshots and a lot more. Let’s create a basic structure for it:

zfs create trunk/lxc -o mountpoint=/srv/lxc

And combine it with LXC:

$ cat /etc/lxc/lxc.conf
lxc.rootfs.backend = zfs
lxc.lxcpath = /srv/lxc
lxc.bdev.zfs.root = trunk/lxc
versable@plombyr:~$ cat /etc/lxc/default.conf
lxc.net.0.type = veth
lxc.net.0.link = br0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx

lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1

Testing the Setup #

Let’s fire up a test container and check if we didn’t mess up the network settings:

$ lxc-create better-damn-work -B zfs --template download -- \
    --dist debian --release bookworm --arch amd64
Using image from local cache
Unpacking the rootfs

---
You just created a Debian bookworm amd64 (20250222_05:24) container.

To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.

Start it and check the IP (It might take a couple of seconds until it gets the IPs assigned):

$ lxc-start better-damn-work && sleep 5 && lxc-ls -f better-damn-work
NAME             STATE   AUTOSTART GROUPS IPV4           IPV6                                UNPRIVILEGED
better-damn-work RUNNING 1         -      192.168.44.154 fde0:d31d:2035:0:216:3eff:fea7:46c4 false

I highly recommend digging in the Getting Started Page from https://linuxcontainers.org to get a grasp of fiddling with LXC.

Continue reading Home Lab: Setting up the Blog Backend.

Footnotes #

  1. In the “Software selection” section of the installation, only “SSH server” and “standard system utilities” were selected. ↩︎

  2. It really feels like the containers are separate machines when the network is integrated that way, and it eliminates the need for routing or using a proxy. ↩︎