Setting up a Private Kubernetes Cluster on Your Own VPS Using Microk8s

Discover the cheapest and easiest way I've found to create personal Kubernetes clusters.

This blog post provides a step-by-step guide to setting up a private Kubernetes cluster on a Virtual Private Server (VPS), with the following features ready and configured:

• VPS firewall setup to expose HTTP/HTTPS & kubeapi server
• MetalLB configured for load balancing and exposing VPS IP
• Ingress and cert-manager setup for quick & automatic TLS management
• Kubeapi server configured for restricted remote access

The prerequisites for this tutorial are: A VPS with a Public IP, A domain with a configurable DNS.

Step 1: Create a base user

Connect to your VPS. In a terminal, create a user and add them to the sudoers.

adduser <user-name>
usermod -aG sudo <user-name>

Step 2: Create SSH keys

On your local machine, create SSH keys to access the server. Set a secure password for the key.

ssh-keygen -b 2048 -t rsa
chmod 600 <key-file> <key-file>.pub

Copy the content of <key-name>.pub to your VPS at /home/<user-name>/.ssh/authorized_keys.

Now we edit the ssh config to require the ssh key, edit /etc/ssh/sshd_config

PasswordAuthentication no # yes before
PubkeyAuthentication yes # no before

Then restart the ssh service service ssh restart

Note: these are minimal ssh setup setups and depending on the environment extra steps should be taken to make ssh more robust agains outside attacks.

Step 3: Set up and enable the firewall

Set up the firewall to allow SSH, HTTP, HTTPS connections and also expose port 16443 which will be used to access the Kubeapi server.

ufw allow ssh
ufw allow http
ufw allow https
ufw allow 16443
ufw enable

Now disconnect from your server and reconnect as the new user:

ssh -i "<private-server-key>" <user-name>@<server-IP>

Step 4: Install and set up microk8s

sudo snap install microk8s --classic --channel=1.28
sudo microk8s enable rbac
sudo microk8s start
microk8s enable ingress dns hostpath-storage cert-manager

If you want to use the current user to manage microk8s; this is not recomended for production clusters

sudo usermod -a -G microk8s <your-user>

Now we create a default cluster issuer: letsencrypt-prod:

microk8s kubectl apply -f - <<EOF
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-prod
spec:
 acme:
   email: <your email>
   server: https://acme-v02.api.letsencrypt.org/directory
   privateKeySecretRef:
     name: lets-encrypt-priviate-key
   solvers:
   - http01:
       ingress:
         class: public
EOF

We also need to enable some additional rules for the firewall (see this github comment):

sudo ufw allow in on cali+
sudo ufw allow out on cali+

To check for general configration issues use microk8s inspect This can give you some easy setup configuration infos and warnings such as:

To enable ip forwarding:

sudo iptables -P FORWARD ACCEPT
sudo apt-get install iptables-persistent

No to konfigure kubeserver access through the host dns, edit /var/snap/microk8s/current/certs/csr.conf.template:

[ alt_names ]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster
DNS.5 = kubernetes.default.svc.cluster.local
DNS.6 = k8s.<your-domain>

Refresh the server certificate:

sudo microk8s refresh-certs --cert server.crt

View the updated client certificate: microk8s config > kubeconfig.yaml. Open the kubeconfig.yaml and edit the server: entry.

...
- cluster:
    certificate-authority-data: XXXXXXXXCENSOREDXXXXXXX
    server: <pub-IP> <---- put k8s.<domain> here
...

Now you can connect to the Kubernetes cluster from your local machine:

KUBECONFIG="./kubeconfig.yaml" kubectl get pods --all-namespaces

Step 5: Setting Up MetalLB

First, prepare <your-domain>. Create a specific or one wildcard DNS entry:

<your-domain> -> <vps-ip>
*.<your-domain> -> <vps-ip>

Enable metallb via microk8s enable metallb:<your-ip>-<your-ip>.

To make metallb work correctly, apply the patch described in this GitHub comment.

#!/bin/sh

command -v kubectl > /dev/null 2>&1 && KUBECTL=kubectl
command -v microk8s.kubectl > /dev/null 2>&1 && KUBECTL=microk8s.kubectl

INGTMPFILE=$(mktemp -t ingress_daemonset.XXXXXXXX)

trap "rm -f ${INGTMPFILE}" 0 1 2 3

${KUBECTL} -n ingress get daemonset nginx-ingress-microk8s-controller -o yaml | \
    sed -e 's|- --publish-status-address=.*|- --publish-service=$(POD_NAMESPACE)/ingress|' > ${INGTMPFILE}

${KUBECTL} diff -f ${INGTMPFILE}
if [ $? -eq 0 ]; then
    echo "No changes need to be made"
else
    ${KUBECTL} apply -f ${INGTMPFILE}
fi

You can now use LoadBalancer nodes that distribute the <VPS-IP>.

Some Tips

Enabling tab completion for microk8s

vim ~/.bash_aliases

# add the fllowing
alias kubectl='microk8s kubectl'
export LC_ALL=en_US.utf-8
export LANG=en_US.utf-8
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)  

Patching External Traffic Policies

kubectl patch svc nodeport -p '{"spec":{"externalTrafficPolicy":"Local"}}'

Testing the preservation of client IP adresses

kubectl expose deployment source-ip-app --name=loadbalancer --port=80 --target-port=8080 --type=LoadBalancer

Conclusion

You now have a Kubernetes cluster with services that can be exposed via ingress, have a load balancer, and cert manager ready. You should be able to access these services through a host domain.

The possibilities are endless from here ;^)

You even have the option to connect this to multiple VPSs to extend your cluster. More on that in a future blog post.