
Alexander is a writer, traveler, and serial digital entrepreneur. He’s the founder of the Tushi, the Gen X personal growth newsletter Further, and offers small business advice at Unemployable.

Twitter

Linkedin

Facebook

Director and Chief Evangelist

Alexander Hipp


Anil is a content-obsessed marketer who loves driving organic traffic for outstanding companies. He is also the creator of Hack the Content, co-host of The Runner, and owner of several digital commerce sites.

Managing Partner

Anil Vugels


Monika is a writer, musician, and content strategist with a passion for language. When she isn’t helping clients, you can find her writing songs, hiking in a northern forest, or furiously journaling.

Director of Operations

Ann Monika

Mastodon

Github

Programmer / Founder / Student

Tim Schupp

A hands-on experiment building a self-managed at-home AI cluster with k3s, Ollama, and LiteLLM.

Building a private LLM Cluster


> Discover the cheapest and easiest way I've found to create personal Kubernetes clusters.

This blog post provides a step-by-step guide to setting up a private Kubernetes cluster on a Virtual Private Server (VPS), with the following features ready and configured:

- VPS firewall setup to expose HTTP/HTTPS & kubeapi server
- MetalLB configured for load balancing and exposing VPS IP
- Ingress and cert-manager setup for quick & automatic TLS management
- Kubeapi server configured for restricted remote access

The **prerequisites** for this tutorial are: **A VPS with a Public IP**, **A domain with a configurable DNS**.

## Step 1: Create a base user

Connect to your VPS. In a terminal, create a user and add them to the sudoers.

First change the root user password to something strong `passwd`

```bash
adduser <user-name>
usermod -aG sudo <user-name>
```

## Step 2: Create SSH keys

On your local machine, create SSH keys to access the server.
Set a secure password for the key.

```bash
ssh-keygen -b 2048 -t rsa
chmod 600 <key-file> <key-file>.pub
```

Copy the content of `<key-name>.pub` to your VPS at `/home/<user-name>/.ssh/authorized_keys`.

Now we edit the ssh config to require the ssh key, edit `/etc/ssh/sshd_config`

```bash
PasswordAuthentication no # yes before
PubkeyAuthentication yes # no before
PermitRootLogin no # Before yes
```

Then restart the ssh service `service ssh restart`

> Note: these are minimal ssh setup setups and depending on the environment extra steps should be taken to make ssh more robust agains outside attacks.

## Step 3: Set up and enable the firewall

Now disconnect from your server and reconnect as the new user:

```bash
ssh -i "<private-server-key>" <user-name>@<server-IP>
```

Set up the firewall to allow SSH, HTTP, HTTPS connections and also expose port `16443` which will be used to access the Kubeapi server.

```bash
sudo ufw allow ssh
sudo ufw allow http
sudo ufw allow https
sudo ufw allow 16443
sudo ufw enable
```

## Step 4: Install and set up microk8s

```bash
sudo snap install microk8s --classic --channel=1.35 # Check for current version!
# sudo microk8s enable rbac (optional) for more fine-grained access control, might influence some following steps
sudo microk8s enable ingress 
sudo microk8s enable dns 
sudo microk8s enable hostpath-storage
sudo microk8s enable cert-manager
sudo microk8s enable metrics-server
sudo microk8s start
```

If you want to use the current user to manage microk8s; this is not recomended for production clusters

```bash
sudo usermod -a -G microk8s <your-user>
```

Now we create a default cluster issuer: `letsencrypt-prod`:

```bash
microk8s kubectl apply -f - <<EOF
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-prod
spec:
 acme:
   email: <your email>
   server: https://acme-v02.api.letsencrypt.org/directory
   privateKeySecretRef:
     name: lets-encrypt-priviate-key
   solvers:
   - http01:
       ingress:
         class: trafiek
EOF
```

> Note that compared to the last guide we are using 'trafiek' as default ingress provider

We also need to enable some additional rules for the firewall ([see this github comment](https://github.com/canonical/microk8s/issues/2418#issuecomment-877350375)):

```bash
sudo ufw allow in on cali+
sudo ufw allow out on cali+
```

To check for general configration issues use `microk8s inspect`
This can give you some easy setup configuration infos and warnings such as:

To enable ip forwarding:

```
sudo iptables -P FORWARD ACCEPT
sudo apt-get install iptables-persistent
```


No to konfigure kubeserver access through the host dns, edit `/var/snap/microk8s/current/certs/csr.conf.template`:

```bash
[ alt_names ]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster
DNS.5 = kubernetes.default.svc.cluster.local
DNS.6 = k8s.<your-domain>
```

Refresh the server certificate:

```bash
sudo microk8s refresh-certs --cert server.crt
```

View the updated client certificate: `microk8s config > kubeconfig.yaml`.
Open the `kubeconfig.yaml` and edit the `server:` entry.

```bash
...
- cluster:
    certificate-authority-data: XXXXXXXXCENSOREDXXXXXXX
    server: <pub-IP> <---- put k8s.<domain> here
...
```

Now you can connect to the Kubernetes cluster from your local machine:

```bash
KUBECONFIG="./kubeconfig.yaml" kubectl get pods --all-namespaces
```

## Step 5 (optional) setup tailscale

```bash
curl -fsSL https://tailscale.com/install.sh | sh && sudo tailscale up --auth-key=<your-tailscale-auth-key>
```

2. Create a tailscale service for that cluster
3. Serve the new kubernetes node

```bash
sudo tailscale serve --bg --service=svc:<your-service-name> --tcp 16433 tcp://127.0.0.1:16433
Serve started and running in the background.
To disable the proxy, run: tailscale serve --service=svc:<your-service-name> --tcp=16433 off
To remove config for the service, run: tailscale serve clear svc:<your-service-name>
```

4. Add the tailscale service IP also to `/var/snap/microk8s/current/certs/csr.conf.template` ( & `sudo microk8s refresh-certs --cert server.crt` )
5. (optinal) close the kubernetes port `ufw block 16433`

## Some Tips

### Chore: Check micok8s certificate expiry!

```bash
sudo microk8s refresh-certs -c
```
bad example:

```bash
sudo microk8s refresh-certs -c
The CA certificate will expire in 2859 days.
The server certificate will expire in -19 days.
The front proxy client certificate will expire in -19 days.
```

refresh them:

```bash
sudo microk8s refresh-certs -e server.crt
sudo microk8s refresh-certs -e front-proxy-client.crt
```

#### Enabling tab completion for microk8s

`vim ~/.bash_aliases`

```bash
# add the fllowing
alias kubectl='microk8s kubectl'
export LC_ALL=en_US.utf-8
export LANG=en_US.utf-8
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)  
```

#### Patching External Traffic Policies

```bash
kubectl patch svc nodeport -p '{"spec":{"externalTrafficPolicy":"Local"}}'
```

#### Testing the preservation of client IP adresses

```bash
kubectl expose deployment source-ip-app --name=loadbalancer --port=80 --target-port=8080 --type=LoadBalancer
```

## Conclusion

You now have a Kubernetes cluster with services that can be exposed via ingress, have a load balancer, and cert manager ready. You should be able to access these services through a host domain. 

> The possibilities are endless from here ;^)

You even have the option to connect this to multiple VPSs to extend your cluster. More on that in a future blog post.


Setup a Microk8s node on only VPS (updated)

Microk8s; self host Kubernetes on the edge


<img width="1906" height="917" alt="OpenCode blog cover" src="/static/assets/opencode-blog-cover.png" />

[Opencode](https://opencode.ai/) is a pretty cool and neatly composable terminal based llm coding agent.
For a project at [tb-software](https://timschupp.de/) and also private personal use in general I'm especially interesting in the usage of open-code or similar tools with local llms, hosted on own hardware. See [my article on running and testing local llms on a 128gb strix halo setup here](https://blog.t1m.me/blog/local-llms-on-strix-halo-128gb-shared-ram).
This article will describe my personal setup as well as some hints how to manage it using a nix-flake - this functions more as future reference - than direct instructions. But it may help someone configure open-code for local llm usage.

### Configuring Open-Code

Just create a `opencode.json` like this:

```json
{
  "$schema": "https://opencode.ai/config.json",
  "provider": { ... },
}
```
Then start adding your providers, fistly the local llm one, e.g.: you could use here [ollama]() though there are probaly some better and more performant options out there ( _these I still have to test again / with newer models_ ). But project like localai lammacpp, ... come to mind. Anything that host api compatible with the openai api schema really.

```json
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "ollama-local",
      "options": {
        "baseURL": "http://localhost:11434/v1"
      },
      "models": { ... }
```
e.g.: to test the models from [my latest benchark](https://blog.t1m.me/blog/local-llms-on-strix-halo-128gb-shared-ram) i added. _Of course you will have to pull the models and corretly setup and start the local lmm server first ( check my other post for an example using ollama on vulkan + nix )._

```json
        "qwen3-coder-next": { "name": "qwen3-coder-next" },
        "mdq100/Qwen3-Coder-30B-A3B-Instruct:30b": { "name": "qwen-code-instuct-small"},
        "nemotron-3-nano": { "name": "nemotron-3-nano" },
        "glm-4.7-flash": { "name": "glm-4.7-flash" },
        "gpt-oss:120b": { "name": "gpt-oss:120b" }
```

<img width="1906" height="917" alt="OpenCode post starting prompt example" src="/static/assets/open-code-post-starting-prompt.png" />

### Usability / Review

As already stated in my benchmark post I was impressed by the performance and usability of the small llms, especially on small simple task like file editing, moving things around, manipulating issues, making simple commits and more. *So it is definitely somewhat hopeful that these open models do seems to perform for some use-case*; So I would say with care and good guardrails they can applied for certain tasks; especially when fully private data processing is required.

Also for the full potential of open-code and priming agents to be able to perform certain tasks in certain environments; you should definitely checkout and learn [how open-code mcps](https://opencode.ai/docs/mcp-servers/) and [agent skills](https://opencode.ai/docs/skills/) are set-up. I my experience the small local models perform best with easy; repeatable and easy to verify tasks - especially when given the right context clues and tools to use - but tasks that involve steps hard to statically automate ( e.g.: text based decision; complex semi autonomous task; browser use; search etc... ). Also it is possible to build in restrictive confirmation based tools for oversight.

### External Providers

Also if you want to use other remove non local providers to test oss models, you could add the following. _Note the api keys are taken from environment you don't need to include them in the config_.

#### 1. Deepinfra

```json
    "deepinfra": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "DeepInfra",
      "options": {
        "baseURL": "https://api.deepinfra.com/v1",
        "apiKeyEnv": "DEEPINFRA_API_KEY"
      },
      "models": { ... }
```

#### 2. groq

```json
    "groq": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Groq",
      "options": {
        "baseURL": "https://api.groq.com/openai/v1",
        "apiKeyEnv": "GROQ_API_KEY"
      },
      "models": { ... }
```

#### a complete config example

```json
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama (local)",
      "options": {
        "baseURL": "http://localhost:11434/v1"
      },
      "models": {
        "qwen3-coder-next": { "name": "qwen3-coder-next" },
        "mdq100/Qwen3-Coder-30B-A3B-Instruct:30b": { "name": "qwen-code-instuct-small"},
        "nemotron-3-nano": { "name": "nemotron-3-nano" },
        "glm-4.7-flash": { "name": "glm-4.7-flash" },
        "gpt-oss:120b": { "name": "gpt-oss:120b" }
      }
    },
    "deepinfra": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "DeepInfra",
      "options": {
        "baseURL": "https://api.deepinfra.com/v1",
        "apiKeyEnv": "DEEPINFRA_API_KEY"
      },
      "models": {
        "nvidia/Nemotron-3-Nano-30B-A3B": { "name": "nvidia/Nemotron-3-Nano-30B-A3B" },
        "moonshotai/Kimi-K2.5": { "name": "moonshotai/Kimi-K2.5" },
        "Qwen/Qwen3-235B-A22B-Instruct-2507": { "name": "Qwen/Qwen3-235B-A22B-Instruct-2507" },
        "MiniMaxAI/MiniMax-M2.5": { "name": "MiniMaxAI/MiniMax-M2.5" }
      }
    },
    "groq": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Groq",
      "options": {
        "baseURL": "https://api.groq.com/openai/v1",
        "apiKeyEnv": "GROQ_API_KEY"
      },
      "models": {
        "moonshotai/kimi-k2-instruct": { "name": "moonshotai/kimi-k2-instruct" },
        "qwen/qwen3-32b": { "name": "qwen/qwen3-32b" }
      }
    }
  }
}
```

### Managing a global open-code install via nix

<img width="1906" height="917" alt="OpenCode performing git push" src="/static/assets/open-code-performing-git-push.png" />

To mange my local open-code installation and configuration I've setup a simple nix flake, in a git sub-module that stores and pulls my open-code configuration and makes it globally available on configuration update.

```nix
{
  description = "Install opencode.json as a global OpenCode config";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs }:
    let
      opencodeJsonPath = ./opencode.json;

      commonOptions = { lib, ... }:
        {
          options.opencodeOllamaLocalSetup = {
            enable = lib.mkOption {
              type = lib.types.bool;
              default = false;
              description = "Whether to install the OpenCode config file globally.";
            };
          };
        };

      nixosModule = { config, lib, ... }:
        let
          cfg = config.opencodeOllamaLocalSetup;
          defaultUsers = lib.attrNames (lib.filterAttrs (_: user:
            (user.isNormalUser or false) || ((user.uid or null) != null && user.uid >= 1000)
          ) config.users.users);
          targetUsers = if cfg.users == null then defaultUsers else cfg.users;
          existingUsers = lib.filter (name: lib.hasAttr name config.users.users) targetUsers;
          userRules = lib.concatLists (map (name:
            let
              user = config.users.users.${name};
              homeDir = user.home or "/home/${name}";
              groupName = user.group or name;
            in [
              "d ${homeDir}/.config/opencode 0700 ${name} ${groupName} -"
              "L+ ${homeDir}/.config/opencode/opencode.json - ${name} ${groupName} - ${opencodeJsonPath}"
            ]
          ) existingUsers);
        in
        {
          options.opencodeOllamaLocalSetup.users = lib.mkOption {
            type = lib.types.nullOr (lib.types.listOf lib.types.str);
            default = null;
            description = "Users to receive ~/.config/opencode/opencode.json (null = all normal users).";
          };

          config = lib.mkIf cfg.enable {
            systemd.tmpfiles.rules = userRules;
          };
        };

      homeManagerModule = { config, lib, ... }:
        let
          cfg = config.opencodeOllamaLocalSetup;
        in
        {
          config = lib.mkIf cfg.enable {
            xdg.configFile."opencode/opencode.json".source = opencodeJsonPath;
          };
        }
    in
    {
      nixosModules.default = { ... }: {
        imports = [ commonOptions nixosModule ];
      };
      homeManagerModules.default = { ... }: {
        imports = [ commonOptions homeManagerModule ];
      };
    }
}
```

<img width="1906" height="917" alt="OpenCode blog section edit example" src="/static/assets/opencode-blog-section-edit-example.png" />


Testing and configuring opencode for usage with local llms

Opencode; Usability with Local LLMs on iGPU w 128GB vram: My Tests


## Local LLMs on Strix Halo; [AMD Ryzen™ AI MAX+ 395](https://www.amd.com/en/blogs/2025/amd-ryzen-ai-max-395-processor-breakthrough-ai-.html) with 128GB Shared Ram: My Tests

LLMs are sometimes stupid, sometimes impressive, often unpredictable and almost never private.
Bigger models are generally hosted in clouds and proprietary, sometimes you can limit access or censor data presented to model providers but almost never you can restrict access fully or be really sure how and when your data is being used.

These are just some of the reasons why you might want to run models locally. But hardware especially graphics cards that can run these models are very pricey and the vram requirement for the more 'mid-range' models just exceeds what is possible with local models.

For some time Apple had an alternative to expensive graphics cards with their m-chip lineup allowing shared VRAM with integrated graphics and CPU, but the prices for higher RAM Mac configurations are still ridiculously high. 
But now there is a 'cheaper' option available for consumers. For a [tb-software](https://timschupp.de/) project I tested the highest available 128GB RAM configuration of a mini PC with the new strix-halo chip to test it and its ability for running local LLMs.

This article reports our early findings and summarizes some configuration steps and the measurement setup I used.
To this point I've tested 5 LLMs and more tests are coming. I also investigated the use and usability of local agents with OpenCode in a companion post: [Opencode; Usability with Local LLMs on iGPU w 128GB VRAM](https://blog.t1m.me/blog/opencode-with-local-llms).

### Setting up ollama with vulkan

First I needed to allocate VRAM to the integrated graphics card, I have not played with the 'dynamic' setting here yet. ( _I have now tried extending VRAM to 94GB more below_ )
For these tests I have chosen to set the VRAM to `65536 MiB` so there is about `67 GB` RAM still available to the CPU. We can also test influence on load speed and inference of varying the available amount of VRAM in the future.

<img width="1451" height="234" alt="Image" src="/static/assets/strix-halo-vram.png" />

Tests were run on NixOS and to access all the libraries required to use the Ollama Vulkan backend I had to install some additional packages and libraries, specifically I added `vulkan-loader` to the NixOS system libraries and ran Ollama with `OLLAMA_LLM_LIBRARY=vulkan OLLAMA_VULKAN=1 ollama serve`. I used `amdgpu_top` to monitor the VRAM and graphics usage ( _It feels like there are still some performance gains to be made, also VRAM is not fully utilized, especially by the smaller models_ ).

### The Test Setup

To test the LLMs for now I just wanted to know basic results for load speed and completion speeds.
I've created a simple fork of [ollama benchmark](https://github.com/aidatatools/ollama-benchmark) ( here [my fork without telemetry](https://github.com/tbscode/tims-ollama-bench-fork) ) to measure the results based on a running Ollama backend.

If you had the exact same hardware, system and configurations, tests could be replicated via:

```bash
python3 -m venv venv
source venv/bin/activate
pip install git+https://github.com/tbscode/tims-ollama-bench-fork.git
tims_llm_benchmark run --custombenchmark=<your-model-config>.yml
```

With models file:

```yaml
file_name: "<config-name>.yml"
version: 2.0.custom
models:
  - model: "<model-name>"
```

And run them all via

```bash
tims_llm_benchmark run --custombenchmark=model-configs/qwen3-coder-next.yaml
tims_llm_benchmark run --custombenchmark=model-configs/nemotron-3-nano.yaml
tims_llm_benchmark run --custombenchmark=model-configs/glm-4.7-flash.yaml
tims_llm_benchmark run --custombenchmark=model-configs/gpt-oss-120b.yaml
tims_llm_benchmark run --custombenchmark=model-configs/qwen-3.5-122b.yaml
```

<img width="1906" height="917" alt="Image" src="/static/assets/ollama-benchmark.png" />

### Test Results

Based on this benchmark and personal prompts and agents tests I've created this table to summarize the results.
There are many more tests to be done and likely also performance improvements through improved configuration, these are just my initial findings and a preliminary review. *But I can definitely say that there are functional local models out there that can definitely complete simple tasks locally on this hardware!*

| Name | Size (GB) | Quality | Load Speed | Tokens / sec |
|--|--|--|--|--|
| **qwen3-coder-next** | 61GB | 👾👾👾👾👾 | Fast | `35.094` |
| **nemotron-3-nano** | 34GB | 👾👾👾👾 | Very Fast | `63.566` |
| **glm-4.7-flash** | 40GB | 👾👾👾👾 | Slow | `50.098` |
| **gpt-oss:120b** | 70GB | 👾👾👾👾 | Very Slow | `31.532` |
| **qwen-3.5:122b** | 81GB | 👾👾👾👾 | Very Slow | `19.158` |
| **qwen-3.5:9b** | 6.6GB | 👾👾👾👾 | Very Slow | `29.52` |

All these models are usable for smaller and simpler local agentic and coding tasks, involving basic files manipulations and tool calls e.g.: via open-code. I was especially impressed by how well `qwen3-coder-next` performed, and also `nemotron-3-nano` that was incredibly fast and smart for such a small thinking model.

### Testing new qwen-3.5 with increased 94gb vram

Now I've updated the bios to allocate 94GB VRAM to allow larger models to run entirely in VRAM.

<img width="1451" height="234" alt="BIOS VRAM settings for 94GB allocation" src="/static/assets/strix_halo_increase_94gb_vram.png" />

Here's the active usage with 94GB VRAM allocated:

<img width="1906" height="917" alt="94GB VRAM active usage showing model loaded in VRAM" src="/static/assets/strix_halo_94gb_vram_active_usage_strix_halo.png" />

With the increased VRAM, I was able to download and run qwen-3.5:95b entirely in VRAM:

<img width="1451" height="917" alt="Downloading qwen-3.5:95b model with Ollama" src="/static/assets/ollama_downloading_qwen122b.png" />


Strix Halo 128GB RAM, 100% local LLM agents, my tests

Local LLMs on Strix Halo 128GB Shared Ram: My Tests


I have been testing several "Agentic" code editors to understand their current capabilities and limitations. This article compares **Antigravity** (Google), **Cursor** (Anysphere), **Windsurf**, and the open-source combination of **VSCodium + Continue**.

## The Contenders

### Antigravity (Google)
Antigravity is Google's entry into the agentic coding space, leveraging their compute and model capabilities. It integrates well for complex, multi-step reasoning tasks.

**Notes:**
- **Pricing:** No official pricing model exists yet.
- **Rate Limits:** The current rate limits are low and can interrupt workflow during intensive sessions.

![Antigravity Screenshot](/static/assets/agentic_editors/antigravity_screenshot.png)

### Cursor
Cursor is a fork of VS Code built by Anysphere. It includes a "Cursor Tab" feature for predictive edits.

**Cloud Agents:**
Cursor's agents operate in a sandboxed cloud environment. They can execute terminal commands, edit files, and create/refine Pull Requests. This is useful for repetitive, dynamic tasks. For example, I used it to integrate new automatic emails for **Little World**, where the agent edited configuration files across multiple locations and restructured code, replacing a previously manual process.

**Pricing:**
The Pro plan costs approximately **€20/month**. However, it includes a limited number of "fast" agent edits; exceeding this limit results in throttling or requires purchasing usage tokens.

![Cursor Screenshot](/static/assets/agentic_editors/cursor_screenshot.png)

### Windsurf
Windsurf is another editor in this space, emphasizing a "Flow" state. Some of the original engineers behind Windsurf are now employed at Google.

### Codium + Continue
For a fully open-source stack, **VSCodium** (a telemetry-free VS Code build) combined with the **Continue** extension allows the use of own models (e.g., Claude 3.5 Sonnet, GPT-4o) while maintaining data control.

![Codium Screenshot](/static/assets/agentic_editors/codium_screenshot.png)

**Trade-offs:**
- **Privacy:** No telemetry or data egress without explicit configuration.
- **Usability:** Major vendors (like Microsoft) do not support non-official builds. Proprietary extensions like Pylance/Pylint are often disabled or broken, requiring workarounds.

[Read my guide on setting up Codium + Continue here](https://timschupp.de/blog/self-hosted-ai-code-editing)

## Verdict

These tools are useful for specific workflows, particularly repetitive tasks involving consistent steps (e.g., adding a database field, updating the API, and modifying the frontend).

### Reality Check

They are not perfect. In my experience, they produce a significant amount of incorrect output. My acceptance rate is approximately **20-30%** of the suggested code. Even accepted code often requires rewriting to match specific requirements. They function as assistants, not replacements.

## Python Tooling Note

The official **Pylint extension for VS Code is proprietary** (part of the Pylance/Python bundle) and is disabled on non-official IDEs like VSCodium.

If using an editor other than the official VS Code or Cursor, I recommend **[basedpyright](https://github.com/DetachHead/basedpyright)** as an open-source alternative.


A comparison of Antigravity, Cursor, Windsurf, and Codium + Continue for agentic coding tasks.

Comparing Agentic Code Editors


Here’s how I keep AI-assisted coding fully under my control: Continue.dev with Codium, pointed at APIs I trust and models I host. Everything stays local or on endpoints I pick - no mistrius data blackholes, no pricy APIs with uncontrolled token usages. Just own hardware and local storage.

### Setting up the local model

`~/.continue/config.yaml`

```yaml
name: My Setup
version: 1.0.0
schema: v1

models:
  - name: "Tims Kimi"
    provider: openai
    model: moonshotai/Kimi-K2-Thinking
    apiBase: https://api.deepinfra.com/v1/openai
    apiKey: <DEEPINFRA_API_KEY>
    roles:
      - chat
      - edit
```

You can even use Google Models outside of Antrigravity:

```yaml
  - name: Gemini 3 Native
    provider: gemini
    model: gemini-3-pro-preview
    apiKey: <GOOGLEAI_API_KEY>
```

### Using self hosted models

There are several tools you can use here; the easiest is probably Ollama with its OpenAI-compatible backend.

e.g.: start qwen coder:

```bash
ollama run qwen3-coder:30b
```

Then include it in your contine.dev configuration:

```yaml
  - name: "Ollama self hosted"
    provider: openai
    model: qwen3-coder:30b
    apiBase: http://localhost:11434/v1
    roles:
      - chat
      - edit
```

And now you can edit the code completely privately using a 100% local model.


### Fixing Python Linting

Microsoft sometimes is a little b**ch, they block the pylint plugin from working with non-official VS Codes.
So I recommend using 'basedpyright'; it works with Codium, and honestly in some aspects is even better than pylint.


A comprehensive guide to setting up fully self-hosted AI code editing with Codium and Continue.dev, keeping your code and AI interactions completely private and under your control.

Latest Posts

Building a private LLM Cluster

Microk8s; self host Kubernetes on the edge

Opencode; Usability with Local LLMs on iGPU w 128GB vram: My Tests

Local LLMs on Strix Halo 128GB Shared Ram: My Tests

Comparing Agentic Code Editors

100% Open-Source Self-Hostable AI Code Editing: Codium & Continue.dev