
Alexander is a writer, traveler, and serial digital entrepreneur. He’s the founder of the Tushi, the Gen X personal growth newsletter Further, and offers small business advice at Unemployable.

Twitter

Linkedin

Facebook

Director and Chief Evangelist

Alexander Hipp


Anil is a content-obsessed marketer who loves driving organic traffic for outstanding companies. He is also the creator of Hack the Content, co-host of The Runner, and owner of several digital commerce sites.

Managing Partner

Anil Vugels


Monika is a writer, musician, and content strategist with a passion for language. When she isn’t helping clients, you can find her writing songs, hiking in a northern forest, or furiously journaling.

Director of Operations

Ann Monika

Mastodon

Github

Programmer / Founder / Student

Tim Schupp

A hands-on experiment building a self-managed at-home AI cluster with k3s, Ollama, and LiteLLM.

Building a private LLM Cluster


Here’s how I keep AI-assisted coding fully under my control: Continue.dev with Codium, pointed at APIs I trust and models I host. Everything stays local or on endpoints I pick - no mistrius data blackholes, no pricy APIs with uncontrolled token usages. Just own hardware and local storage.

### Setting up the local model

`~/.continue/config.yaml`

```yaml
name: My Setup
version: 1.0.0
schema: v1

models:
  - name: "Tims Kimi"
    provider: openai
    model: moonshotai/Kimi-K2-Thinking
    apiBase: https://api.deepinfra.com/v1/openai
    apiKey: <DEEPINFRA_API_KEY>
    roles:
      - chat
      - edit
```

You can even use Google Models outside of Antrigravity:

```yaml
  - name: Gemini 3 Native
    provider: gemini
    model: gemini-3-pro-preview
    apiKey: <GOOGLEAI_API_KEY>
```

### Using self hosted models

There are several tools you can use here; the easiest is probably Ollama with its OpenAI-compatible backend.

e.g.: start qwen coder:

```bash
ollama run qwen3-coder:30b
```

Then include it in your contine.dev configuration:

```yaml
  - name: "Ollama self hosted"
    provider: openai
    model: qwen3-coder:30b
    apiBase: http://localhost:11434/v1
    roles:
      - chat
      - edit
```

And now you can edit the code completely privately using a 100% local model.


### Fixing Python Linting

Microsoft sometimes is a little b**ch, they block the pylint plugin from working with non-official VS Codes.
So I recommend using 'basedpyright'; it works with Codium, and honestly in some aspects is even better than pylint.


A comprehensive guide to setting up fully self-hosted AI code editing with Codium and Continue.dev, keeping your code and AI interactions completely private and under your control.

Self Hosting

Building a private LLM Cluster

100% Open-Source Self-Hostable AI Code Editing: Codium & Continue.dev