Reza Ghafari

Local Environment Kubernetes Options

It is always fun to have a local Kubernetes to try new things on but there are a few different options with varying capabilities that might look confusing at first.

Some of the following products are also useful for setting up a home lab, home server or even production grade IoT.

In this short summary which is basically a copy-paste of my personal notes, we look at different options that will help developers work with local Kubernetes clusters.

This is obviously ignoring cloud providers. If you have access to a cloud provider they all now provide k8s as service or you can set it up yourself on their VMs.

Disclaimer: These tools evolve too fast hence the validity of this material might be short but I try to keep it up to date!

Kind - Started in 2019, Kind is a mini Kubernetes on Docker by the Kubernertes developers. The cluster is basically a docker container running on docker runtime

  • Pros: Very fast startup
  • Cons: high memory usage, not a fully compliant Kubernetes
  • MultiNode: yes - 1 master
  • MultiCluster: yes
  • Min Memory: 8 GB

K3s and K3d - Lightweight, fully compliant ks8 which is best for IoT, ARM and edge developed by Rancher. k3d is a lightweight wrapper to run k3s in docker.

  • Pros: fully compliant, production grade, very light
  • Cons: Not on Windows yet
  • MultiNode: yes
  • Min Memory: 100 MB

Docker Desktop - If you use Docker desktop on Mac or Windows you get a k8s cluster for free. It runs on docker.

  • Pros: Images are readily available, comes as part of docker desktop
  • Cons: high memory usage
  • MultiNode: no
  • Min Memory: 4 GB

Minikube - Mini k8s by the Kubernetes developers that can run on a VM (e.g. VirtualBox) or even docker. It started as early as k8s itself around 2012.

  • Pros: Simplicity, add-on based for adding extra functionality as needed e.g. loadbalancer
  • Cons: Running on VM is slow
  • MultiNode: no
  • Multicluster: no
  • Min Memory: 2GB

MicroK8s - Fully compliant k8s by Canonical (Ubuntu folks) that is supported on ARM, Windows, Mac, etc. It was previously only available with Snap package manager but it has improved much.
It’s lightweight and great for IoT similar to K3s. It also supports a number of extension add-ons similar to Minikube.

  • Pros: Runs on any platform, production grade, has extensions
  • Cons: On Windows/Mac need a VM
  • MultiNode: yes
  • Min Memory: 4GB

FireKube - This is opensource by Weave that runs on VMs. Haven’t tried yet!

Virtual Machines

If you want to setup a multi node clusters in a home server you would need VMs and this discussion will not be complete if we don’t talk about different ways of spinning VMs.

This is in my order of preference:

LXD, LXC - This is the fun one. Runs only on linux but it’s the fastest and lightest approach since it uses userspace concept of linux. All VMs share the same kernel as host though. LXC/D runs the full OS system and provides an environment similar to a VM but docker runs an application process in a container.

Multipass - By Canonical, runs on any platform, good startup time, nice command line. On Windows/Mac needs a hypervisor such as Vbox or hyper-v or even docker which is faster.

Vagrant - By HashiCorp, runs on any platform, ok startup time, good command line. Needs a hypervisor such as hyper-v, vbox or docker (faster)

Virtualbox/Parallel - Simple to use, UI based, command line sucks. Need to tweak for best memory usage, etc.