young green seedlings sprouting from brown soil-filled compartments in a planting tray
Just like starting off your tomato fruits in a starter tray, you can preseed Debian for a standardised install﹡

The centerpieces of my homelab, next to a new Optiplex 3050 SFF, a Banana Pi R3 or a Mikrotik RBM11G, are my 3 HPE Microserver Gen8. I already wrote a few words about them and you should really check them out if you have one yourself. But today is less about them specifically and more about how I automated the provisioning of them with Debian preseeding.

When I started my homelab, I started off by manually installing Debian on all the microservers, then installing Proxmox and clustering them together, with the whole ceph setup. This was interesting and fun, but I soon realized that having only 16GB of RAM wouldn’t allow much in terms of VMs, as well as giving up soo much storage for ceph. So I went back on the ceph cluster and went with ZFS which would allow me to only loose roughly 25% instead of 66% of my storage capacity. I also noticed that having one big cluster didn’t allow me to play around as much as I wanted. To top that off, I have never been a big VM guy, but rather a Kubernetes guy, so I have a whole of two VMs right now: one running K3s and another running a samba share.

So I dismantled the cluster and allowed myself to play around with the microservers individually. I decided to have one “production” machine, that would run essential services like homeassistant, NVR or Jellyfin, while the other two would be open for either storage or testing.

The problem

Having a playground also means getting to reinstall the machine quite often, just like a Zen garden that you rake every day. After the 3rd install you quickly get tired of the same 15 installation options you have to click through and even though you do everything the same, it always feels slightly off.

I wanted to have the possibility to just get the same starting ground every time, without having to click through the installer, as well as having the feeling that all 3 machines are setup identically and correctly.

The options

This led me to look into bare metal provisioning automation. I quickly was led to Ironic, which has bifrost and Metal³ by a friend, but I quickly realized that the increased complexity of those tools was in no relation to the amount of machines I had to provision.

Since I only use Debian as my preferred Linux OS, I was roughly aware of the possibility to preseed a Debian installer, to automate the installation process. It looked very archaic and the documentation is not as helpful as I would like to, but I decided to give it a try.

Fuck around and find out

As the good programmer that I am, I started of with asking ChatGPT to create a preseed.cfg for my microservers. Which worked reasonably well until I needed to validate the file: Start a microserver (which takes 2 minutes or more), pxe boot into netboot.xyz, start the Debian installer with the preseed file from a http server, fail, reboot, rinse, repeat…

After a while (3 tries exactly) I was fed up with waiting for the microservers and called it a day, everything seemed to be so overly complicated and the documentation felt so devoid of any sense. I went for a 2 week trip, so the whole topic would have to wait either way.

Coming back from the trip, I decided to start from scratch. I already found the example-preseed.txt from Debian and decided to go from there. I decided for the following setup:

  • English Language, Spanish Timezone, German Keyboard
  • Only root user with a predefined password
  • Use /dev/sdf as a OS disk (which is the ODD drive on the HPE Microserver Gen8)
  • Put Grub on /dev/sda (which is the SD Card on the HPE Microserver Gen8)
  • Set the kernel parameters for iLo to work correctly
  • Enable security, backports and automatic updates
  • Allow SSH root login with a password

Initially I also wanted to setup the storage raid, but I quickly realized that this is not possible with the preseed file and would also be a bit overkill.

What I learned

I learned that if you want to use Debian preseed files to automate your installation, you should keep it simple, very simple. There are some basic setup possibilities that you can use when setting up packages, but I would recommend to keep it as simple as possible and use tools like Ansible afterwards.

I also learned that there is a “bug” in the way the OpenSSH-server is being installed and configured. You would think that when I set openssh-server/permit-root-login boolean true that I would be allowed to login as root via SSH aka PermitRootLogin yes. Well you are wrong, as some other Debian maintainers already found out in 2016. But even setting openssh-server/permit-root-login boolean false doesn’t work, for some magical reason, that I wasn’t willing to explore anymore.

In the end I used a solution that I found elsewhere, which will automatically change the PermitRootLogin in the /etc/ssh/sshd_config file after the installation is done.

d-i preseed/late_command string \
    in-target sed -i 's/^.*PermitRootLogin.*/PermitRootLogin yes/g' /etc/ssh/sshd_config

After having found a template that works for me and my microservers, I am now able to just start the microserver, pxe boot into the Debian installer and let it do it’s thing.

The full preseed file

You can find the full preseed file here

Next steps

One more optimization step I am thinking about is to create a script that will automatically create a upto-date netboot Debian image with the preseed file included, which will decrease the manual steps I have to do even further.

Another thing will be to learn Ansible and setup a playbook that will do the rest of the setup: LCAP, ZFS and other things that need to be done for a “basic” setup.

* Photo from Markus Spiske auf Unsplash