6 minutes
Migrating a single node k3s cluster
The intro to the migration
Two years ago I was running my private kubernetes cluster using microK8s on a Hetzner cloud server. My introduction into kubernetes was roughly three years ago, when we decided at mecodia to use kubernetes for our new PaaS Platform. After getting to know Kubernetes and passing the steep learning kurve, we were confident to move from unmanaged unrelated servers running docker to running a coherent Kubernetes cluster (with the support of Rancher). I also decided that I wanted to run my own private cluster, for everything I have.
When I first started, the best solution to do so without running a full kubernetes was microK8s, which is a single node kubernetes cluster that runs on a single cloud machine. It was long a dream of mine to move to a proper dedicated machine, but the cost of a new dedicated machine was too high for me at the time. Thankfully Hetzner has a server auction where you can get a dedicated server for a fraction of the price, if you are willing to take a machine that has been used before.
So in September 2019 I was able to get a nice Xeon E3-1246 v3 with 32GB of DDR3 RAM and 2x 256GB SSDs for roughly 40€ a month. Moving from the cloud server to the dedicated server was a breeze, since I was able to just manually move the little data I had from the cloud server to the dedicated server and then switch the DNS records. I also made the decision to move from microK8s to k3s, since k3s was a lot more lightweight and easier to manage, as well as providing a default CSI for local storage.
For an extension to storage I used a storage box (1TB for 3.50€ a month) and the csi-driver-smb to mount the storage box into the cluster. This is mostly for backups and long term storage, as the storage box is quite slow and limited by the 1Gbit/s network connection.
Fast-forward to today, I am still running the same server, I have put more services onto the cluster: Mail, Git, Grafana, Prometheus, this website, etc. Still the server isn’t breaking a sweat in terms of CPU and RAM usage, but the storage is getting a bit tight. With only 256GB of Sata SSD storage, I am neither winning a space nor a speed contest. So when I was just browsing the Hetzner server auction, I saw a nice Xeon E3-1275 v6 with 64GB of DDR4 RAM and 2x 512GB NVMe SSDs for a 5€ cheaper 35€ a month, which made me immediately pull the trigger.
🚨⚠️ This is not a guide on how to migrate a k3s cluster, this is just a documentation of how I migrated my single node k3s cluster from one Hetzner dedicated server to another. If you are unsure test this on a cloud machine first, as I did. YMMV
The migration of the data
Now the question was, how do I migrate the data from the old server to the new server? A simple node migration is not possible with k3s, since single node k3s doesn’t support etcd snapshots, when you are running a SQLite database as the datastore. Another question was the local storage, since I was using the local-path-provisioner, which is a CSI driver that uses the local filesystem to provide storage. The CSI has no way to move data from one server to the other server on it’s own. So I had to come up with a solution to move the data from the old server to the new server. I came up with the following solution:
0. Prepare the new server
Install all the things you need on the new server, e.g. packages, unattended-upgrades, etc. Don’t install k3s yet.
Make absolutely sure that the new server has the same hostname as the old server, otherwise the local-path-provisioner and other parts of k3s will be confused and not work.
1. Put the existing server into rescue mode
This way we make sure the k3s cluster is not running and we can access the filesystem, without any data being modified during the move. You can also mount the filesystem into mount and chroot into it, as documented by Hetzner here.
2. Rsync the data
I used rsync to move the data from the old server to the new server. I used the following command to move the data:
# Migrate the server datastore
rsync -avvz --progress /var/lib/rancher/k3s/server/ root@new-server:/var/lib/rancher/k3s/server/
# Migrate the server local storage
rsync -avvz --progress /var/lib/rancher/k3s/storage/ root@new-server:/var/lib/rancher/k3s/storage/
You may need to set up an SSH key to be able to log in to the new server without a password.
Don’t forget any other configuration files you may have changed, e.g. /etc/rancher/k3s/config.yaml.
or /etc/rancher/k3s/manifests/.
3. Install K3s in the new server
Install k3s on the new server using the following command:
curl -sfL https://get.k3s.io | sh -
Use any parameters you used on the old server, e.g. --cluster-cidr= or --service-cidr=. and also copy any manifests
you used.
4. Change the DNS records
Change the DNS records to point to the new server. This will make sure that any new requests will be served by the new server.
5. Check if everything is working
Check if everything is working as expected. If not, you can always switch back to the old server by changing the DNS records back. Make sure all services are running as expected and will come up after a reboot.
Reset up any monitoring you may have, e.g. Prometheus, Grafana, etc. and any backup solutions you may have, e.g. Velero, Borgbackup and make sure they are working.
The cleanup of the old server
When and if you are sure you have everything migrated, you can shut down the old server and cancel the contract with
# Wipe the discs
dd if=/dev/zero of=/dev/sda bs=1M
dd if=/dev/zero of=/dev/sdb bs=1M
# or use the wipe or shred command
wipe /dev/sda
shred /dev/sdb
The conclusion
All in all it was way less painful than I had expected. I was able to migrate the server in a few hours and had everything up and running in a few more hours. The biggest problem was that there was actually very little documentation on how to migrate a single node k3s cluster, so I had to figure out a lot of things on my own. You can find the resources I found down below.
Helpful resources
﹡ Photo from Barth Bailey on Unsplash