Building a private LLM Cluster

Building a private LLM cluster

This post covers my hands-on experiment of building a local, self-managed AI cluster at a reasonable price point for at-home AI/LLM usage.

Intro

I like using LLMs and exploring their capabilities; I'm not very fond of the direction the industry is taking.

But I don't think the development in the 'AI/LLM' space will be halted, although I do think there is a direction that is better for society overall.

That is the future where good small and performant AI models are available for everyone to host themselves on their own compute. Keeping all the data private and allowing for much better encapsulation and sandboxing of LLM workloads.

Considerations for building an AI cluster

First, we need hardware. As seen in my initial OS LLM tests, I have access to a basic Bosgame mini PC with the new Ryzen AI 395 chip. After those initial tests, and considering current RAM rarity and prices, I extended my setup with another Bosgame mini PC. This also allows me to test and use a more complex local multi-node Kubernetes setup.

Thus for this article we will use 2x AMD Strix Halo 128GB RAM mini PCs; we will allocate both to have 94GB VRAM in BIOS; and we will leave the remaining RAM for backend and Kubernetes resources and other services.

Since the setup gets more complex, I prefer not to rely on Docker Compose alone and instead choose an orchestration tool that I can also manage externally through an API. For me, that choice is Kubernetes; and for system configuration management I opted to use k3s, instead of my usual microk8s.

Now we have solutions for hardware ✅ and clustering software ✅. Both of the nodes currently run in a home-subnetwork with their own router ✅. But since I plan to set up a multi-region cluster in the future, I am already extending this setup to work across networks and across LAN/WLAN.

For that, we will use Tailscale in this setup; I'm also planning to build a more static setup that will work with WireGuard only in the future; but for now we choose convenience.

Still, there are more missing parts:

1. multi-node LLM backend and orchestration
2. load balancing and deployment

So for 1. I'll choose a multi-node Ollama Helm chart install; the initial aim is to have two nodes with the same models available so they can be load-balanced. For 2. I'll choose LiteLLM as it's fairly easy to configure, supports API authorization, and supports the OpenAI API schema; and has a bunch of handy load balancing and routing features.

Experiment Setup

For easy configuration and management of both nodes, I've connected both to KVMs. I can control, observe, and manage both nodes simply from my laptop; this is important as I do not have enough screens and keyboards, and I also don't want to manually set up all nodes.

Here is a picture of the wiring; I've also configured a simple hyprland setup with screen mirroring at my display resolution to allow me to easily view the screen on KVM and another monitor.

System Set-Up

To configure and manage the cluster, I want to avoid extensive manual setup while still keeping the setup repeatable and orchestratable from outside. Thus I chose to install NixOS on both nodes; in my NixOS setup I can easily set up SSH access keys, my tailnet, and everything else in a central deterministic configuration.

Notably I ensure each system runs

• NixOS, amdgpu, k3s ( see below )

And of course all required basic system tooling:

• git, vim, k9s, kubectl, helm etc...

Also I've configured basic SSH keys for access to my nodes from inside my tailscale network. Similarly, I've created a setup that lets me pre-define hosts per system. This will be important later when we obtain DNS-01 certificates to resolve, for example, our LiteLLM dashboard through our tailnet.

Here you can see my two KVM control tabs open, one selected and showing current VRAM and CPU usage through btop and amdgpu.

K3s Cluster Set-Up

In networking, I chose to route cluster traffic over the tailnet to allow me to have mini PCs connected via LAN and WLAN, to be able to move them across location and use them across networks in the future.

I configured one cluster node to be the k3s agent and the other to be the server. As said I've opted to connect over the tailnet directly, the cluster configuration is managed directly through nix; this generally equates to roughly this k3s command:

Wait for the services to become available and retrieve the kubeconfig.yaml from the main k8s node.

Setting up DNS-01 challenges and certificate issuing behind NAT

Note: this part will differ based on your DNS provider

To access the LiteLLM API in the browser conveniently, we want to have a real certificate!

Example DNS challenge set-up:

• https://github.com/fabmade/cert-manager-webhook-ionos

Create API secret ionos-secret.yaml

Create a production cluster issuer letsencrypt-ionos-prod-account-key:

Ollama Multi-Node Installation

First we need to teach our cluster to work with our integrated graphics, then we can install ollama with vulkan enabled and run llms.

Set up k3s AMD ROCm resources

Now we have two devices with AMD integrated GPUs and Kubernetes installed. But we can see that the Kubernetes cluster doesn't yet have AMD GPU resources listed on nodes:

If GPU plugin is not installed yet, the AMD column is usually empty or <none>.

Now we can install the ROCm GPU plugin:

Now we can see the GPUs listed:

Set up Ollama Helm chart

I opted to use this Helm chart. Add the Helm chart:

We create some basic values that allow us to replicate the setup 1:1 on our other node:

Note that we apply these values twice with a minimal modification:

vs

This ensures we have identical Ollama instances running on both hosts, but this also means that we always need to make sure both Ollama instance node states are synced.

LiteLLM load balancing and routing setup

I chose to use this Helm chart as a base setup for LiteLLM.

We want our traffic to be routed based on system usage, i.e. when one mini PC is under NPU load it should receive less traffic. Optimally we also want the API to block traffic if the system is under so much load that the experience would be too bad (more on that later).

Setup basic chart values:

Now, since we have set up initial cluster DNS-01 challenges, we can request an official Let's Encrypt certificate via:

This will make it much easier to access our LiteLLM UI through the browser as it won't complain about SSL and self-signed certificates. But note you still need to register <your-domain> in your own /etc/hosts as we did NOT create a public DNS entry. (Shout-out to JannisT for this setup suggestion.)

Then start by adding your models. Depending on node setup, you need to register model endpoints; for me it's one per cluster node, so two. We choose to throttle requests depending on model size, this configuration will be further adjusted and tested in future blog posts.

Now finally we also want to configure the router setup. I opted to generate a general PROXY_MASTER_KEY instead of individual keys per endpoint, just for benchmarking for now.

Configure LiteLLM to also route external providers

I found it very convenient to also use LiteLLM to route paid external providers, as this gives me a central location to manage access and tokens. As I've also run my benchmark on some public providers for comparison, this was also set up, e.g. one could add this to integrate OpenAI gpt-5.5 or gpt-5.4-mini:

This can also be used very conveniently to manage LLM access, e.g. in a company. And finally we can install the chart:

Now we can retrieve the master key:

Setup complete. Now we can see all our clusters and nodes running, and we're ready for benchmarking.

Benchmark

Compared to the last benchmark, I wanted to extend the test range. Specifically, I'm interested in finding out:

• TTFT (time to first token); and
• TTFT dependency on context length

To reset the tests, I always ensure that all models are killed and fully unloaded, and I restart the Ollama containers every time. ./scripts/kill_all_running_completions.sh --force-restart wraps a Kubernetes exec command into a simple script that can be hooked into my updated Ollama benchmark. I also refactored the benchmark to work in 'API only' mode so I can compare my results with known existing LLM hosters (for example OpenAI).

I also observed that it is very important to preload the Ollama model, because TTFT is significantly reduced when the model is already loaded into memory. After observing this, I had to re-run all tests because the initial TTFT values would significantly drag down the overall average.

Benchmark Results 1: Model Comparison

In the first version of this article, I tested the following models:

• deepseek-r1:1.5b
• deepseek-r1:14b
• deepseek-r1:32b
• deepseek-r1:7b
• deepseek-r1:8b
• gemma4-31b
• gemma4:26b
• gemma4:e4b
• nemotron-cascade-2:30b
• qwen3.6:27b
• qwen3.6:35b
• extra: gpt-5.4-mini
• extra: gpt-5.4-nano
• extra: gpt-5.5

Mainly due to already having spent extended time setting up the cluster and benchmarking, and also being rate-limited by Ollama for model downloads after some time...

Actually I re-ran the benchmark with updated settings, see further below for improved results

1) Basic benchmark comparison

2) Context dependency matrix (cell = TTFT ms / TPS)

Legend: each context cell is TTFT(ms) / tokens_per_second.

Blog Addition Benchmark Re-Runs

Re-run bigger default context and flash attention:

• set OLLAMA_FLASH_ATTENTION=1
• set OLLAMA_KV_CACHE_TYPE=q8_0
• set OLLAMA_CONTEXT_LENGTH=32768

1) Basic benchmark comparison

2) Context dependency matrix (cell = TTFT ms / TPS)

Re-Run different KV Cache

• OLLAMA_FLASH_ATTENTION=1
• OLLAMA_KV_CACHE_TYPE=f16
• OLLAMA_CONTEXT_LENGTH=32768
• OLLAMA_KEEP_ALIVE=1h
• OLLAMA_NUM_PARALLEL=1
• OLLAMA_MAX_LOADED_MODELS=1

1) Basic benchmark comparison

2) Context dependency matrix (cell = TTFT ms / TPS)

After using some of the OSS LLMs for my workflows, I realized: Maybe my context length test is wrong, because in a normal chat session you continue with the exact previous context. In normal usage I experienced TTFT much lower than what I was seeing in my benchmarking. So I rewrote the context size test to actually emulate a "continuing" session. But then I also realized that cache reuse is only possible if the same request gets routed to the same node. So even after these changes, the initial settings performed best; I'll test further variations in future experiments.

Re-Run Re-using old context on long context re-runs

1) Basic benchmark comparison

2) Context dependency matrix (cell = TTFT ms / TPS)

Legend: each context cell is TTFT(ms) / tokens_per_second.

Cross-setting TTFT context dependency (selected model families)

This table compares TTFT (ms) only across context sizes for gemma4-*, nemotron-cascade-*, and qwen3-* over the different benchmark settings.

So after that change, TTFT still stayed roughly the same (but now we know we need some sort of cache-hit / node-reuse strategy in the future).

That must be it for now, the article is way too long already. I'm starting to actively use my own hosted models for different things and will report back further in the future.

Cheers Tim