
Alexander is a writer, traveler, and serial digital entrepreneur. He’s the founder of the Tushi, the Gen X personal growth newsletter Further, and offers small business advice at Unemployable.

Twitter

Linkedin

Facebook

Director and Chief Evangelist

Alexander Hipp


Anil is a content-obsessed marketer who loves driving organic traffic for outstanding companies. He is also the creator of Hack the Content, co-host of The Runner, and owner of several digital commerce sites.

Managing Partner

Anil Vugels


Monika is a writer, musician, and content strategist with a passion for language. When she isn’t helping clients, you can find her writing songs, hiking in a northern forest, or furiously journaling.

Director of Operations

Ann Monika

Mastodon

Github

Programmer / Founder / Student

Tim Schupp


I use this pattern when I want to keep workloads private in a k3s cluster, but still expose selected services to the public internet through a separate public microk8s edge cluster.

This is a full, production-style walkthrough for exposing a private service across two clusters with Tailscale Kubernetes Operator.

It is tailored to this exact topology:

- Source (private) cluster: k3s, kubeconfig at `kubeconfig-node-A.yaml`
- Public edge cluster: microk8s, kubeconfig at `kubeconfig-node-B.yaml`
- Private app example: service `<app-service>` in k3s namespace `<app-namespace>`
- Public hostname example: `example.server.com`
- TLS issuer on public cluster: `letsencrypt-prod` using ACME HTTP-01

---

## 1. Architecture and Traffic Flow

Goal:

- Keep app workloads private in k3s.
- Publish them safely via public ingress in microk8s.
- Use Tailscale as encrypted inter-cluster transport.

Data path:

1. Client requests `https://example.server.com`.
2. DNS resolves to the public microk8s node IP.
3. microk8s ingress controller (`IngressClass=public`) receives request.
4. Ingress routes to `<app-service>-tailnet` service in microk8s.
5. Tailscale operator egress proxy in microk8s forwards traffic over tailnet.
6. Tailnet reaches the k3s Tailscale-exposed service `<app-hostname-node-a>.<tailnet>.ts.net`.
7. k3s service forwards to the private app pod.

This pattern works for many NAT/internal services, not only one specific app.

---

## 2. Prerequisites

### 2.1 DNS

- `example.server.com` must point to the public microk8s ingress IP.

### 2.2 cert-manager in public cluster

- `cert-manager` installed and healthy.
- `ClusterIssuer` named `letsencrypt-prod` configured for HTTP-01.

Check:

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl get pods -n cert-manager
kubectl get clusterissuer letsencrypt-prod
```

### 2.3 Ingress controller class in microk8s

In this environment, ingress class is `public` (not `nginx`).

Check:

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl get ingressclass
kubectl -n ingress get pods
```

### 2.4 Tailscale operator installed in both clusters

Check:

```bash
export KUBECONFIG=./kubeconfig-node-A.yaml
kubectl get pods -n tailscale

export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl get pods -n tailscale
```

Both should show the operator pod running.

---

## 3. Phase A: Expose Service in k3s (Source Cluster)

You can expose an existing `ClusterIP` service via annotations.

For service `<app-service>` in namespace `<app-namespace>`:

```bash
export KUBECONFIG=./kubeconfig-node-A.yaml
kubectl annotate service <app-service> -n <app-namespace> \
  tailscale.com/expose=true \
  tailscale.com/hostname=<app-hostname-node-a> \
  --overwrite
```

Verify:

```bash
export KUBECONFIG=./kubeconfig-node-A.yaml
kubectl -n <app-namespace> get svc <app-service> -o yaml
kubectl -n tailscale get pods,svc,secrets
```

Expected:

- The source service has `tailscale.com/expose: "true"`.
- A proxy pod appears in `tailscale` namespace (for example `ts-<app-service>-xxxxx-0`).
- A tailnet FQDN is created.

Get tailnet FQDN from secret:

```bash
export KUBECONFIG=./kubeconfig-node-A.yaml
kubectl -n tailscale get secret <proxy-secret-name> -o jsonpath='{.data.device_fqdn}' | base64 -d
```

Example value:

- `<app-hostname-node-a>.taild4f875.ts.net.`

Keep this FQDN. It is the inter-cluster target.

---

## 4. Phase B: Configure microk8s as Public Edge Cluster

Use Tailscale cluster egress mode in microk8s. This is the critical part.

Do not use a plain `ExternalName` directly to `<name>.ts.net` without Tailscale egress annotations. The ingress controller may fail DNS resolution and return `504`.

### 4.1 Manifest

Create `app-proxy.yaml` in the public cluster context with this content:

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: <app-namespace>
---
apiVersion: v1
kind: Service
metadata:
  name: <app-service>-tailnet
  namespace: <app-namespace>
  annotations:
    tailscale.com/tailnet-fqdn: <app-hostname-node-a>.taild4f875.ts.net
spec:
  type: ExternalName
  externalName: placeholder
  ports:
    - name: http
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: <app-service>-public
  namespace: <app-namespace>
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: public
  tls:
      - hosts:
        - example.server.com
      secretName: example-server-com-tls
  rules:
    - host: example.server.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: <app-service>-tailnet
                port:
                  number: 80
```

Apply:

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl apply -f app-proxy.yaml
```

Why `externalName: placeholder` is correct:

- Tailscale operator rewrites `spec.externalName` to an internal service it creates in `tailscale` namespace.
- This enables reliable egress proxying to tailnet targets.

---

## 5. Validation Workflow (End-to-End)

Run these checks in order.

### 5.1 In public cluster, check rewritten egress service

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n <app-namespace> get svc <app-service>-tailnet -o yaml
```

Expected:

- `spec.externalName` changed from `placeholder` to something like `ts-<app-service>-tailnet-xxxxx.tailscale.svc.cluster.local`.
- `status.conditions` includes `TailscaleProxyReady=True`.

### 5.2 Check tailscale proxy objects in public cluster

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n tailscale get pods,svc,secrets
```

Expected:

- pod `ts-<app-service>-tailnet-xxxxx-0` is `Running`.

### 5.3 Check ingress and certificate

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n <app-namespace> get ingress <app-service>-public -o wide
kubectl -n <app-namespace> get certificate,certificaterequest,order,challenge
```

Expected:

- Ingress class shown as `public`.
- Certificate `example-server-com-tls` is `Ready=True`.

### 5.4 In-cluster backend connectivity test

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n <app-namespace> run curlcheck --image=curlimages/curl:8.8.0 --restart=Never --rm -i --command -- sh -c "curl -sS -o /dev/null -w '%{http_code}\n' http://<app-service>-tailnet.<app-namespace>.svc.cluster.local"
```

Expected output:

- `200`

### 5.5 Public test

```bash
curl -I https://example.server.com
```

Expected:

- HTTP `200` (or app-specific expected code), valid TLS chain.

---

## 6. Troubleshooting by Symptom

### 6.1 Symptom: `504 Gateway Time-out`

Most common causes:

- Ingress points to wrong class (`nginx` instead of `public` in this microk8s setup).
- Service is direct `ExternalName -> <name>.ts.net` without `tailscale.com/tailnet-fqdn` egress annotation.
- Tailscale egress proxy pod not created or not ready.

Check ingress logs:

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n ingress logs <nginx-ingress-controller-pod> --tail=200
```

If you see repeated DNS `name error` for tailnet hostname or `placeholder`, your service wiring is incorrect or still converging.

### 6.2 Symptom: TLS cert not issued

Check:

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n <app-namespace> get certificate,certificaterequest,order,challenge
kubectl -n <app-namespace> describe ingress <app-service>-public
```

Common causes:

- `example.server.com` DNS not pointing to public ingress endpoint.
- Wrong `cluster-issuer` name.
- HTTP-01 solver path blocked by firewall/proxy.

### 6.3 Symptom: Ingress serves default cert

Cause:

- `spec.tls[].secretName` mismatched with cert-manager-produced secret.

Fix:

- Ensure `Ingress.spec.tls.secretName` equals certificate secret (for example `example-server-com-tls`).

### 6.4 Symptom: Tailscale operator ignores service

Check:

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n tailscale logs deploy/operator --tail=300
```

If logs show no egress reconciliation for your service, ensure:

- Service is type `ExternalName`.
- Has annotation `tailscale.com/tailnet-fqdn` or `tailscale.com/tailnet-ip`.
- Tailscale operator is healthy.

---

## 7. Operational Best Practices

- Use dedicated namespace per proxied application (for example `<app-namespace>`).
- Use explicit hostnames per service (for example `<app1-hostname-node-a>`, `<app2-hostname-node-a>`).
- Keep private cluster service as `ClusterIP` only; avoid public LB there.
- Terminate public TLS in edge cluster ingress.
- Monitor ingress and tailscale operator logs for DNS and proxy readiness conditions.
- Keep cert-manager issuer names consistent across manifests.

---

## 8. Pattern Reuse for Any NAT Service

To expose another private k3s service through the same public microk8s cluster:

1. Annotate source service in k3s with `tailscale.com/expose=true` and a unique hostname.
2. In microk8s, create annotated `ExternalName` service with `tailscale.com/tailnet-fqdn` to that hostname.
3. Create public ingress for the desired DNS host with `ingressClassName: public`.
4. Attach cert-manager annotation `cert-manager.io/cluster-issuer: letsencrypt-prod`.
5. Validate service rewrite, proxy pod readiness, and certificate readiness.

This gives a repeatable, encrypted cross-cluster edge publishing model.

---

## 9. Reference Commands Cheat Sheet

Source cluster (k3s):

```bash
export KUBECONFIG=./kubeconfig-node-A.yaml
kubectl -n <app-namespace> get svc <app-service> -o yaml
kubectl -n tailscale get pods,svc,secrets
```

Public cluster (microk8s):

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n <app-namespace> get svc,ingress
kubectl -n <app-namespace> get certificate,certificaterequest,order,challenge
kubectl -n tailscale get pods,svc,secrets
kubectl -n tailscale logs deploy/operator --tail=200
kubectl -n ingress logs <nginx-ingress-controller-pod> --tail=200
```

Public endpoint test:

```bash
curl -I https://example.server.com
```


A production-style setup to publish private k3s services through a public microk8s edge cluster using the Tailscale Kubernetes Operator.

Expose NAT Services Across Two Kubernetes Clusters with Tailscale


I needed a PostgreSQL setup on Kubernetes that is:

- allows automatic creation of db instances
- allow configuration based pre-defining users, passwords and configurations
- can be included into other helm charts as depencency
- single configuration, one-install, full setup approach

For this, [CloudNativePG Helm Chart](TODO) is one of the cleanest approaches I've found so far ( after the [Bitnami Postgsql Helm Chart being moved closed source]() ).

So this is the exact setup I currently use as a base template.
Everything below is copy/paste-ready and end-to-end reproducible.

### What this creates

- namespace: `app-db`
- cluster: `app-postgres`
- database: `app`
- app user: `appuser`
- app password: `apppass`
- readonly user: `readonly`
- readonly password: `readonlypass`
- table: `public.items`

### 0) Move to your project and export kubeconfig

```bash
export KUBECONFIG=./kubeconfig.yaml
kubectl get nodes
```

### 1) Install the CloudNativePG operator

```bash
helm repo add cnpg https://cloudnative-pg.github.io/charts
helm repo update
helm upgrade --install cnpg cnpg/cloudnative-pg --namespace cnpg-system --create-namespace
kubectl get pods -n cnpg-system
```

Wait until the operator pod is `Running` before continuing.

### 2) Create a manifest folder

```bash
mkdir -p cnpg-app-db
```

### 3) Create namespace manifest

```bash
cat > cnpg-app-db/namespace.yaml <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: app-db
EOF
```

### 4) Create app credentials secret

```bash
cat > cnpg-app-db/db-secret.yaml <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: app-db-user
  namespace: app-db
type: kubernetes.io/basic-auth
stringData:
  username: appuser
  password: apppass
EOF
```

### 5) Create PostgreSQL cluster manifest

```bash
cat > cnpg-app-db/postgres-cluster.yaml <<'EOF'
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: app-postgres
  namespace: app-db
spec:
  instances: 2
  imageName: ghcr.io/cloudnative-pg/postgresql:17
  storage:
    size: 10Gi
  bootstrap:
    initdb:
      database: app
      owner: appuser
      secret:
        name: app-db-user
      postInitApplicationSQL:
        - CREATE USER readonly WITH PASSWORD 'readonlypass';
        - GRANT CONNECT ON DATABASE app TO readonly;
        - CREATE TABLE IF NOT EXISTS public.items (id SERIAL PRIMARY KEY, name TEXT NOT NULL, created_at TIMESTAMPTZ DEFAULT NOW());
        - ALTER TABLE public.items OWNER TO appuser;
        - GRANT ALL PRIVILEGES ON TABLE public.items TO appuser;
        - INSERT INTO public.items (name) VALUES ('seed-row-from-bootstrap');
        - GRANT USAGE ON SCHEMA public TO readonly;
        - GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly;
        - ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO readonly;
EOF
```

### 6) Create full Python test script

```bash
cat > cnpg-app-db/test_db_query.py <<'EOF'
#!/usr/bin/env python3
import argparse
import os
import sys
from urllib.parse import quote_plus

import psycopg


def build_dsn(args: argparse.Namespace) -> str:
    if args.dsn:
        return args.dsn

    user = quote_plus(args.user)
    password = quote_plus(args.password)
    host = args.host
    port = args.port
    dbname = args.database
    return f"postgresql://{user}:{password}@{host}:{port}/{dbname}"


def main() -> int:
    parser = argparse.ArgumentParser(description="Run a test query against CloudNativePG")
    parser.add_argument("--host", default=os.getenv("PGHOST", "127.0.0.1"), help="PostgreSQL host")
    parser.add_argument("--port", default=os.getenv("PGPORT", "5432"), help="PostgreSQL port")
    parser.add_argument("--database", default=os.getenv("PGDATABASE", "app"), help="Database name")
    parser.add_argument("--user", default=os.getenv("PGUSER", "appuser"), help="Database user")
    parser.add_argument(
        "--password",
        default=os.getenv("PGPASSWORD", "apppass"),
        help="Database password",
    )
    parser.add_argument(
        "--dsn",
        default=os.getenv("DATABASE_URL"),
        help="Full PostgreSQL DSN. Overrides host/user/password options when set.",
    )
    args = parser.parse_args()
    dsn = build_dsn(args)

    try:
        with psycopg.connect(dsn) as conn:
            with conn.cursor() as cur:
                cur.execute("SELECT current_user, current_database(), version()")
                user_name, db_name, version = cur.fetchone()
                print(f"connected_as={user_name}")
                print(f"database={db_name}")
                print(f"version={version}")

                cur.execute("SELECT id, name, created_at FROM public.items ORDER BY id")
                rows = cur.fetchall()
                print(f"items_count={len(rows)}")
                for row in rows:
                    print(f"item id={row[0]} name={row[1]} created_at={row[2]}")
    except Exception as exc:
        print(f"connection/query failed: {exc}", file=sys.stderr)
        return 1

    return 0


if __name__ == "__main__":
    raise SystemExit(main())
EOF
```

### 7) Fresh redeploy (safe reset for this cluster)

If you've tested a few times and want a clean restart, run the block below.
This recreates the setup from the manifests above.

```bash
kubectl delete cluster app-postgres -n app-db --ignore-not-found=true
kubectl delete pvc -n app-db --all --ignore-not-found=true
kubectl delete secret app-db-user -n app-db --ignore-not-found=true
kubectl apply -f cnpg-app-db/namespace.yaml
kubectl apply -f cnpg-app-db/db-secret.yaml
kubectl apply -f cnpg-app-db/postgres-cluster.yaml
```

### 8) Wait until cluster is ready

```bash
kubectl wait --for=condition=Ready cluster/app-postgres -n app-db --timeout=600s
kubectl get cluster -n app-db
kubectl get pods -n app-db
kubectl get svc -n app-db
```

Expected result:

- `app-postgres` shows healthy and ready instances
- Pods `app-postgres-1` and `app-postgres-2` are `Running`
- Services exist: `app-postgres-rw`, `app-postgres-ro`, `app-postgres-r`

### 9) Optional direct `psql` test in the pod

```bash
kubectl exec -it -n app-db app-postgres-1 -- psql -U appuser -d app
```

Then run:

```sql
SELECT current_user, current_database();
SELECT * FROM public.items;
\q
```

### 10) Install Python dependency (`psycopg`)

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install psycopg[binary]
```

### 11) Test app user (`appuser`) via RW service

Terminal 1:

```bash
kubectl port-forward -n app-db svc/app-postgres-rw 5432:5432
```

Terminal 2:

```bash
source .venv/bin/activate
python cnpg-app-db/test_db_query.py --host 127.0.0.1 --port 5432 --database app --user appuser --password apppass
```

### 12) Test readonly user (`readonly`) via RO service

Terminal 1:

```bash
kubectl port-forward -n app-db svc/app-postgres-ro 5433:5432
```

Terminal 2:

```bash
source .venv/bin/activate
python cnpg-app-db/test_db_query.py --host 127.0.0.1 --port 5433 --database app --user readonly --password readonlypass
```

### 13) Useful connection strings

```text
postgresql://appuser:apppass@app-postgres-rw.app-db.svc.cluster.local:5432/app
postgresql://readonly:readonlypass@app-postgres-ro.app-db.svc.cluster.local:5432/app
postgresql://appuser:apppass@127.0.0.1:5432/app
postgresql://readonly:readonlypass@127.0.0.1:5433/app
```

### Notes from my side

- Keeping bootstrap SQL in the cluster manifest is very convenient for small/medium setups.
- Splitting `rw` and `ro` service usage early helps avoid accidental write paths later in app code.
- For production, definitely swap the demo passwords and move secrets handling to your preferred secret manager flow.

That's it.
From here you have a reproducible CloudNativePG Postgres setup with:

- predefined users
- schema bootstrap
- seed data
- python connectivity test
- read/write and readonly validation paths

## Optional Add-On: DNS-01 cert flow

If you want to keep using your domain-based access (`test-db.<your-domain>.com`) with stronger trust setup across clients, a DNS-01 cert-manager flow is a good optional add-on.

At minimum:

1. install your DNS webhook solver
2. create a `ClusterIssuer`
3. issue certificate(s) and use them in your service exposure stack

I still keep this optional here, because CNPG already gives working Postgres TLS out of the box.

## Optional Add-On: tiny tailscale gimmick (cluster-to-cluster)

If you have tailscale operator on both clusters, you can expose this DB quickly via tailnet and consume it from another cluster using a simple `ExternalName` service.

High-level mini flow:

1. annotate exposed DB service with `tailscale.com/expose=true`
2. use tailnet hostname (`test-db.<tailnet>.ts.net`)
3. in cluster 2, create `ExternalName` -> that hostname

Useful when you just want quick private cross-cluster connectivity without extra public routing work.

Cheers Tim


A full copy/paste-ready walkthrough to install CloudNativePG, deploy PostgreSQL on Kubernetes, bootstrap users and tables, and verify with Python.

CloudNativePG PostgreSQL End-to-End Setup Guide

A hands-on experiment building a self-managed at-home AI cluster with k3s, Ollama, and LiteLLM.

Building a private LLM Cluster


This blog post provides a step-by-step guide to setting up a private Kubernetes cluster on a Virtual Private Server (VPS), with the following features ready and configured:

- VPS firewall setup to expose HTTP/HTTPS & kubeapi server
- MetalLB configured for load balancing and exposing VPS IP
- Ingress and cert-manager setup for quick & automatic TLS management
- Kubeapi server configured for restricted remote access

The **prerequisites** for this tutorial are: **A VPS with a Public IP**, **A domain with a configurable DNS**.

## Step 1: Create a base user

Connect to your VPS. In a terminal, create a user and add them to the sudoers.

First change the root user password to something strong `passwd`

```bash
adduser <user-name>
usermod -aG sudo <user-name>
```

## Step 2: Create SSH keys

On your local machine, create SSH keys to access the server.
Set a secure password for the key.

```bash
ssh-keygen -b 2048 -t rsa
chmod 600 <key-file> <key-file>.pub
```

Copy the content of `<key-name>.pub` to your VPS at `/home/<user-name>/.ssh/authorized_keys`.

Now we edit the ssh config to require the ssh key, edit `/etc/ssh/sshd_config`

```bash
PasswordAuthentication no # yes before
PubkeyAuthentication yes # no before
PermitRootLogin no # Before yes
```

Then restart the ssh service `service ssh restart`

> Note: these are minimal ssh setup setups and depending on the environment extra steps should be taken to make ssh more robust agains outside attacks.

## Step 3: Set up and enable the firewall

Now disconnect from your server and reconnect as the new user:

```bash
ssh -i "<private-server-key>" <user-name>@<server-IP>
```

Set up the firewall to allow SSH, HTTP, HTTPS connections and also expose port `16443` which will be used to access the Kubeapi server.

```bash
sudo ufw allow ssh
sudo ufw allow http
sudo ufw allow https
sudo ufw allow 16443
sudo ufw enable
```

## Step 4: Install and set up microk8s

```bash
sudo snap install microk8s --classic --channel=1.35 # Check for current version!
# sudo microk8s enable rbac (optional) for more fine-grained access control, might influence some following steps
sudo microk8s enable ingress 
sudo microk8s enable dns 
sudo microk8s enable hostpath-storage
sudo microk8s enable cert-manager
sudo microk8s enable metrics-server
sudo microk8s start
```

If you want to use the current user to manage microk8s; this is not recomended for production clusters

```bash
sudo usermod -a -G microk8s <your-user>
```

Now we create a default cluster issuer: `letsencrypt-prod`:

```bash
microk8s kubectl apply -f - <<EOF
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-prod
spec:
 acme:
   email: <your email>
   server: https://acme-v02.api.letsencrypt.org/directory
   privateKeySecretRef:
     name: lets-encrypt-priviate-key
   solvers:
   - http01:
       ingress:
         class: trafiek
EOF
```

> Note that compared to the last guide we are using 'trafiek' as default ingress provider

We also need to enable some additional rules for the firewall ([see this github comment](https://github.com/canonical/microk8s/issues/2418#issuecomment-877350375)):

```bash
sudo ufw allow in on cali+
sudo ufw allow out on cali+
```

To check for general configration issues use `microk8s inspect`
This can give you some easy setup configuration infos and warnings such as:

To enable ip forwarding:

```
sudo iptables -P FORWARD ACCEPT
sudo apt-get install iptables-persistent
```


No to konfigure kubeserver access through the host dns, edit `/var/snap/microk8s/current/certs/csr.conf.template`:

```bash
[ alt_names ]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster
DNS.5 = kubernetes.default.svc.cluster.local
DNS.6 = k8s.<your-domain>
```

Refresh the server certificate:

```bash
sudo microk8s refresh-certs --cert server.crt
```

View the updated client certificate: `microk8s config > kubeconfig.yaml`.
Open the `kubeconfig.yaml` and edit the `server:` entry.

```bash
...
- cluster:
    certificate-authority-data: XXXXXXXXCENSOREDXXXXXXX
    server: <pub-IP> <---- put k8s.<domain> here
...
```

Now you can connect to the Kubernetes cluster from your local machine:

```bash
KUBECONFIG="./kubeconfig.yaml" kubectl get pods --all-namespaces
```

## Step 5 (optional) setup tailscale

```bash
curl -fsSL https://tailscale.com/install.sh | sh && sudo tailscale up --auth-key=<your-tailscale-auth-key>
```

2. Create a tailscale service for that cluster
3. Serve the new kubernetes node

```bash
sudo tailscale serve --bg --service=svc:<your-service-name> --tcp 16433 tcp://127.0.0.1:16433
Serve started and running in the background.
To disable the proxy, run: tailscale serve --service=svc:<your-service-name> --tcp=16433 off
To remove config for the service, run: tailscale serve clear svc:<your-service-name>
```

4. Add the tailscale service IP also to `/var/snap/microk8s/current/certs/csr.conf.template` ( & `sudo microk8s refresh-certs --cert server.crt` )
5. (optinal) close the kubernetes port `ufw block 16433`

## Some Tips

### Chore: Check micok8s certificate expiry!

```bash
sudo microk8s refresh-certs -c
```
bad example:

```bash
sudo microk8s refresh-certs -c
The CA certificate will expire in 2859 days.
The server certificate will expire in -19 days.
The front proxy client certificate will expire in -19 days.
```

refresh them:

```bash
sudo microk8s refresh-certs -e server.crt
sudo microk8s refresh-certs -e front-proxy-client.crt
```

#### Enabling tab completion for microk8s

`vim ~/.bash_aliases`

```bash
# add the fllowing
alias kubectl='microk8s kubectl'
export LC_ALL=en_US.utf-8
export LANG=en_US.utf-8
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)  
```

#### Patching External Traffic Policies

```bash
kubectl patch svc nodeport -p '{"spec":{"externalTrafficPolicy":"Local"}}'
```

#### Testing the preservation of client IP adresses

```bash
kubectl expose deployment source-ip-app --name=loadbalancer --port=80 --target-port=8080 --type=LoadBalancer
```

## Conclusion

You now have a Kubernetes cluster with services that can be exposed via ingress, have a load balancer, and cert manager ready. You should be able to access these services through a host domain. 

> The possibilities are endless from here ;^)

You even have the option to connect this to multiple VPSs to extend your cluster. More on that in a future blog post.


Setup a Microk8s node on only VPS (updated)

Microk8s; self host Kubernetes on the edge


<img width="1906" height="917" alt="OpenCode blog cover" src="/static/assets/opencode-blog-cover.png" />

[Opencode](https://opencode.ai/) is a pretty cool and neatly composable terminal based llm coding agent.
For a project at [tb-software](https://timschupp.de/) and also private personal use in general I'm especially interesting in the usage of open-code or similar tools with local llms, hosted on own hardware. See [my article on running and testing local llms on a 128gb strix halo setup here](https://blog.t1m.me/blog/local-llms-on-strix-halo-128gb-shared-ram).
This article will describe my personal setup as well as some hints how to manage it using a nix-flake - this functions more as future reference - than direct instructions. But it may help someone configure open-code for local llm usage.

### Configuring Open-Code

Just create a `opencode.json` like this:

```json
{
  "$schema": "https://opencode.ai/config.json",
  "provider": { ... },
}
```
Then start adding your providers, fistly the local llm one, e.g.: you could use here [ollama]() though there are probaly some better and more performant options out there ( _these I still have to test again / with newer models_ ). But project like localai lammacpp, ... come to mind. Anything that host api compatible with the openai api schema really.

```json
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "ollama-local",
      "options": {
        "baseURL": "http://localhost:11434/v1"
      },
      "models": { ... }
```
e.g.: to test the models from [my latest benchark](https://blog.t1m.me/blog/local-llms-on-strix-halo-128gb-shared-ram) i added. _Of course you will have to pull the models and corretly setup and start the local lmm server first ( check my other post for an example using ollama on vulkan + nix )._

```json
        "qwen3-coder-next": { "name": "qwen3-coder-next" },
        "mdq100/Qwen3-Coder-30B-A3B-Instruct:30b": { "name": "qwen-code-instuct-small"},
        "nemotron-3-nano": { "name": "nemotron-3-nano" },
        "glm-4.7-flash": { "name": "glm-4.7-flash" },
        "gpt-oss:120b": { "name": "gpt-oss:120b" }
```

<img width="1906" height="917" alt="OpenCode post starting prompt example" src="/static/assets/open-code-post-starting-prompt.png" />

### Usability / Review

As already stated in my benchmark post I was impressed by the performance and usability of the small llms, especially on small simple task like file editing, moving things around, manipulating issues, making simple commits and more. *So it is definitely somewhat hopeful that these open models do seems to perform for some use-case*; So I would say with care and good guardrails they can applied for certain tasks; especially when fully private data processing is required.

Also for the full potential of open-code and priming agents to be able to perform certain tasks in certain environments; you should definitely checkout and learn [how open-code mcps](https://opencode.ai/docs/mcp-servers/) and [agent skills](https://opencode.ai/docs/skills/) are set-up. I my experience the small local models perform best with easy; repeatable and easy to verify tasks - especially when given the right context clues and tools to use - but tasks that involve steps hard to statically automate ( e.g.: text based decision; complex semi autonomous task; browser use; search etc... ). Also it is possible to build in restrictive confirmation based tools for oversight.

### External Providers

Also if you want to use other remove non local providers to test oss models, you could add the following. _Note the api keys are taken from environment you don't need to include them in the config_.

#### 1. Deepinfra

```json
    "deepinfra": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "DeepInfra",
      "options": {
        "baseURL": "https://api.deepinfra.com/v1",
        "apiKeyEnv": "DEEPINFRA_API_KEY"
      },
      "models": { ... }
```

#### 2. groq

```json
    "groq": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Groq",
      "options": {
        "baseURL": "https://api.groq.com/openai/v1",
        "apiKeyEnv": "GROQ_API_KEY"
      },
      "models": { ... }
```

#### a complete config example

```json
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama (local)",
      "options": {
        "baseURL": "http://localhost:11434/v1"
      },
      "models": {
        "qwen3-coder-next": { "name": "qwen3-coder-next" },
        "mdq100/Qwen3-Coder-30B-A3B-Instruct:30b": { "name": "qwen-code-instuct-small"},
        "nemotron-3-nano": { "name": "nemotron-3-nano" },
        "glm-4.7-flash": { "name": "glm-4.7-flash" },
        "gpt-oss:120b": { "name": "gpt-oss:120b" }
      }
    },
    "deepinfra": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "DeepInfra",
      "options": {
        "baseURL": "https://api.deepinfra.com/v1",
        "apiKeyEnv": "DEEPINFRA_API_KEY"
      },
      "models": {
        "nvidia/Nemotron-3-Nano-30B-A3B": { "name": "nvidia/Nemotron-3-Nano-30B-A3B" },
        "moonshotai/Kimi-K2.5": { "name": "moonshotai/Kimi-K2.5" },
        "Qwen/Qwen3-235B-A22B-Instruct-2507": { "name": "Qwen/Qwen3-235B-A22B-Instruct-2507" },
        "MiniMaxAI/MiniMax-M2.5": { "name": "MiniMaxAI/MiniMax-M2.5" }
      }
    },
    "groq": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Groq",
      "options": {
        "baseURL": "https://api.groq.com/openai/v1",
        "apiKeyEnv": "GROQ_API_KEY"
      },
      "models": {
        "moonshotai/kimi-k2-instruct": { "name": "moonshotai/kimi-k2-instruct" },
        "qwen/qwen3-32b": { "name": "qwen/qwen3-32b" }
      }
    }
  }
}
```

### Managing a global open-code install via nix

<img width="1906" height="917" alt="OpenCode performing git push" src="/static/assets/open-code-performing-git-push.png" />

To mange my local open-code installation and configuration I've setup a simple nix flake, in a git sub-module that stores and pulls my open-code configuration and makes it globally available on configuration update.

```nix
{
  description = "Install opencode.json as a global OpenCode config";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs }:
    let
      opencodeJsonPath = ./opencode.json;

      commonOptions = { lib, ... }:
        {
          options.opencodeOllamaLocalSetup = {
            enable = lib.mkOption {
              type = lib.types.bool;
              default = false;
              description = "Whether to install the OpenCode config file globally.";
            };
          };
        };

      nixosModule = { config, lib, ... }:
        let
          cfg = config.opencodeOllamaLocalSetup;
          defaultUsers = lib.attrNames (lib.filterAttrs (_: user:
            (user.isNormalUser or false) || ((user.uid or null) != null && user.uid >= 1000)
          ) config.users.users);
          targetUsers = if cfg.users == null then defaultUsers else cfg.users;
          existingUsers = lib.filter (name: lib.hasAttr name config.users.users) targetUsers;
          userRules = lib.concatLists (map (name:
            let
              user = config.users.users.${name};
              homeDir = user.home or "/home/${name}";
              groupName = user.group or name;
            in [
              "d ${homeDir}/.config/opencode 0700 ${name} ${groupName} -"
              "L+ ${homeDir}/.config/opencode/opencode.json - ${name} ${groupName} - ${opencodeJsonPath}"
            ]
          ) existingUsers);
        in
        {
          options.opencodeOllamaLocalSetup.users = lib.mkOption {
            type = lib.types.nullOr (lib.types.listOf lib.types.str);
            default = null;
            description = "Users to receive ~/.config/opencode/opencode.json (null = all normal users).";
          };

          config = lib.mkIf cfg.enable {
            systemd.tmpfiles.rules = userRules;
          };
        };

      homeManagerModule = { config, lib, ... }:
        let
          cfg = config.opencodeOllamaLocalSetup;
        in
        {
          config = lib.mkIf cfg.enable {
            xdg.configFile."opencode/opencode.json".source = opencodeJsonPath;
          };
        }
    in
    {
      nixosModules.default = { ... }: {
        imports = [ commonOptions nixosModule ];
      };
      homeManagerModules.default = { ... }: {
        imports = [ commonOptions homeManagerModule ];
      };
    }
}
```

<img width="1906" height="917" alt="OpenCode blog section edit example" src="/static/assets/opencode-blog-section-edit-example.png" />


Testing and configuring opencode for usage with local llms

Opencode; Usability with Local LLMs on iGPU w 128GB vram: My Tests


## Local LLMs on Strix Halo; [AMD Ryzen™ AI MAX+ 395](https://www.amd.com/en/blogs/2025/amd-ryzen-ai-max-395-processor-breakthrough-ai-.html) with 128GB Shared Ram: My Tests

LLMs are sometimes stupid, sometimes impressive, often unpredictable and almost never private.
Bigger models are generally hosted in clouds and proprietary, sometimes you can limit access or censor data presented to model providers but almost never you can restrict access fully or be really sure how and when your data is being used.

These are just some of the reasons why you might want to run models locally. But hardware especially graphics cards that can run these models are very pricey and the vram requirement for the more 'mid-range' models just exceeds what is possible with local models.

For some time Apple had an alternative to expensive graphics cards with their m-chip lineup allowing shared VRAM with integrated graphics and CPU, but the prices for higher RAM Mac configurations are still ridiculously high. 
But now there is a 'cheaper' option available for consumers. For a [tb-software](https://timschupp.de/) project I tested the highest available 128GB RAM configuration of a mini PC with the new strix-halo chip to test it and its ability for running local LLMs.

This article reports our early findings and summarizes some configuration steps and the measurement setup I used.
To this point I've tested 5 LLMs and more tests are coming. I also investigated the use and usability of local agents with OpenCode in a companion post: [Opencode; Usability with Local LLMs on iGPU w 128GB VRAM](https://blog.t1m.me/blog/opencode-with-local-llms).

### Setting up ollama with vulkan

First I needed to allocate VRAM to the integrated graphics card, I have not played with the 'dynamic' setting here yet. ( _I have now tried extending VRAM to 94GB more below_ )
For these tests I have chosen to set the VRAM to `65536 MiB` so there is about `67 GB` RAM still available to the CPU. We can also test influence on load speed and inference of varying the available amount of VRAM in the future.

<img width="1451" height="234" alt="Image" src="/static/assets/strix-halo-vram.png" />

Tests were run on NixOS and to access all the libraries required to use the Ollama Vulkan backend I had to install some additional packages and libraries, specifically I added `vulkan-loader` to the NixOS system libraries and ran Ollama with `OLLAMA_LLM_LIBRARY=vulkan OLLAMA_VULKAN=1 ollama serve`. I used `amdgpu_top` to monitor the VRAM and graphics usage ( _It feels like there are still some performance gains to be made, also VRAM is not fully utilized, especially by the smaller models_ ).

### The Test Setup

To test the LLMs for now I just wanted to know basic results for load speed and completion speeds.
I've created a simple fork of [ollama benchmark](https://github.com/aidatatools/ollama-benchmark) ( here [my fork without telemetry](https://github.com/tbscode/tims-ollama-bench-fork) ) to measure the results based on a running Ollama backend.

If you had the exact same hardware, system and configurations, tests could be replicated via:

```bash
python3 -m venv venv
source venv/bin/activate
pip install git+https://github.com/tbscode/tims-ollama-bench-fork.git
tims_llm_benchmark run --custombenchmark=<your-model-config>.yml
```

With models file:

```yaml
file_name: "<config-name>.yml"
version: 2.0.custom
models:
  - model: "<model-name>"
```

And run them all via

```bash
tims_llm_benchmark run --custombenchmark=model-configs/qwen3-coder-next.yaml
tims_llm_benchmark run --custombenchmark=model-configs/nemotron-3-nano.yaml
tims_llm_benchmark run --custombenchmark=model-configs/glm-4.7-flash.yaml
tims_llm_benchmark run --custombenchmark=model-configs/gpt-oss-120b.yaml
tims_llm_benchmark run --custombenchmark=model-configs/qwen-3.5-122b.yaml
```

<img width="1906" height="917" alt="Image" src="/static/assets/ollama-benchmark.png" />

### Test Results

Based on this benchmark and personal prompts and agents tests I've created this table to summarize the results.
There are many more tests to be done and likely also performance improvements through improved configuration, these are just my initial findings and a preliminary review. *But I can definitely say that there are functional local models out there that can definitely complete simple tasks locally on this hardware!*

| Name | Size (GB) | Quality | Load Speed | Tokens / sec |
|--|--|--|--|--|
| **qwen3-coder-next** | 61GB | 👾👾👾👾👾 | Fast | `35.094` |
| **nemotron-3-nano** | 34GB | 👾👾👾👾 | Very Fast | `63.566` |
| **glm-4.7-flash** | 40GB | 👾👾👾👾 | Slow | `50.098` |
| **gpt-oss:120b** | 70GB | 👾👾👾👾 | Very Slow | `31.532` |
| **qwen-3.5:122b** | 81GB | 👾👾👾👾 | Very Slow | `19.158` |
| **qwen-3.5:9b** | 6.6GB | 👾👾👾👾 | Very Slow | `29.52` |

All these models are usable for smaller and simpler local agentic and coding tasks, involving basic files manipulations and tool calls e.g.: via open-code. I was especially impressed by how well `qwen3-coder-next` performed, and also `nemotron-3-nano` that was incredibly fast and smart for such a small thinking model.

### Testing new qwen-3.5 with increased 94gb vram

Now I've updated the bios to allocate 94GB VRAM to allow larger models to run entirely in VRAM.

<img width="1451" height="234" alt="BIOS VRAM settings for 94GB allocation" src="/static/assets/strix_halo_increase_94gb_vram.png" />

Here's the active usage with 94GB VRAM allocated:

<img width="1906" height="917" alt="94GB VRAM active usage showing model loaded in VRAM" src="/static/assets/strix_halo_94gb_vram_active_usage_strix_halo.png" />

With the increased VRAM, I was able to download and run qwen-3.5:95b entirely in VRAM:

<img width="1451" height="917" alt="Downloading qwen-3.5:95b model with Ollama" src="/static/assets/ollama_downloading_qwen122b.png" />


Strix Halo 128GB RAM, 100% local LLM agents, my tests

Local LLMs on Strix Halo 128GB Shared Ram: My Tests


I have been testing several "Agentic" code editors to understand their current capabilities and limitations. This article compares **Antigravity** (Google), **Cursor** (Anysphere), **Windsurf**, and the open-source combination of **VSCodium + Continue**.

## The Contenders

### Antigravity (Google)
Antigravity is Google's entry into the agentic coding space, leveraging their compute and model capabilities. It integrates well for complex, multi-step reasoning tasks.

**Notes:**
- **Pricing:** No official pricing model exists yet.
- **Rate Limits:** The current rate limits are low and can interrupt workflow during intensive sessions.

![Antigravity Screenshot](/static/assets/agentic_editors/antigravity_screenshot.png)

### Cursor
Cursor is a fork of VS Code built by Anysphere. It includes a "Cursor Tab" feature for predictive edits.

**Cloud Agents:**
Cursor's agents operate in a sandboxed cloud environment. They can execute terminal commands, edit files, and create/refine Pull Requests. This is useful for repetitive, dynamic tasks. For example, I used it to integrate new automatic emails for **Little World**, where the agent edited configuration files across multiple locations and restructured code, replacing a previously manual process.

**Pricing:**
The Pro plan costs approximately **€20/month**. However, it includes a limited number of "fast" agent edits; exceeding this limit results in throttling or requires purchasing usage tokens.

![Cursor Screenshot](/static/assets/agentic_editors/cursor_screenshot.png)

### Windsurf
Windsurf is another editor in this space, emphasizing a "Flow" state. Some of the original engineers behind Windsurf are now employed at Google.

### Codium + Continue
For a fully open-source stack, **VSCodium** (a telemetry-free VS Code build) combined with the **Continue** extension allows the use of own models (e.g., Claude 3.5 Sonnet, GPT-4o) while maintaining data control.

![Codium Screenshot](/static/assets/agentic_editors/codium_screenshot.png)

**Trade-offs:**
- **Privacy:** No telemetry or data egress without explicit configuration.
- **Usability:** Major vendors (like Microsoft) do not support non-official builds. Proprietary extensions like Pylance/Pylint are often disabled or broken, requiring workarounds.

[Read my guide on setting up Codium + Continue here](https://timschupp.de/blog/self-hosted-ai-code-editing)

## Verdict

These tools are useful for specific workflows, particularly repetitive tasks involving consistent steps (e.g., adding a database field, updating the API, and modifying the frontend).

### Reality Check

They are not perfect. In my experience, they produce a significant amount of incorrect output. My acceptance rate is approximately **20-30%** of the suggested code. Even accepted code often requires rewriting to match specific requirements. They function as assistants, not replacements.

## Python Tooling Note

The official **Pylint extension for VS Code is proprietary** (part of the Pylance/Python bundle) and is disabled on non-official IDEs like VSCodium.

If using an editor other than the official VS Code or Cursor, I recommend **[basedpyright](https://github.com/DetachHead/basedpyright)** as an open-source alternative.


A comparison of Antigravity, Cursor, Windsurf, and Codium + Continue for agentic coding tasks.

Comparing Agentic Code Editors


Here’s how I keep AI-assisted coding fully under my control: Continue.dev with Codium, pointed at APIs I trust and models I host. Everything stays local or on endpoints I pick - no mistrius data blackholes, no pricy APIs with uncontrolled token usages. Just own hardware and local storage.

### Setting up the local model

`~/.continue/config.yaml`

```yaml
name: My Setup
version: 1.0.0
schema: v1

models:
  - name: "Tims Kimi"
    provider: openai
    model: moonshotai/Kimi-K2-Thinking
    apiBase: https://api.deepinfra.com/v1/openai
    apiKey: <DEEPINFRA_API_KEY>
    roles:
      - chat
      - edit
```

You can even use Google Models outside of Antrigravity:

```yaml
  - name: Gemini 3 Native
    provider: gemini
    model: gemini-3-pro-preview
    apiKey: <GOOGLEAI_API_KEY>
```

### Using self hosted models

There are several tools you can use here; the easiest is probably Ollama with its OpenAI-compatible backend.

e.g.: start qwen coder:

```bash
ollama run qwen3-coder:30b
```

Then include it in your contine.dev configuration:

```yaml
  - name: "Ollama self hosted"
    provider: openai
    model: qwen3-coder:30b
    apiBase: http://localhost:11434/v1
    roles:
      - chat
      - edit
```

And now you can edit the code completely privately using a 100% local model.


### Fixing Python Linting

Microsoft sometimes is a little b**ch, they block the pylint plugin from working with non-official VS Codes.
So I recommend using 'basedpyright'; it works with Codium, and honestly in some aspects is even better than pylint.


A comprehensive guide to setting up fully self-hosted AI code editing with Codium and Continue.dev, keeping your code and AI interactions completely private and under your control.

100% Open-Source Self-Hostable AI Code Editing: Codium & Continue.dev


As privacy preserving communication options are vanishing rapidly, there remain only a few options for private secure communication.
While messaging services like Signal promise a good level of security and anonymity, its server is still managed by a corporate entity.

To truly control your own privacy and communication, you should consider joining a Matrix server hosted by a trusted party, or consider hosting one yourself.
This blog post will show how you can quickly setup and deploy a Matrix server on Kubernetes using [ananace's Helm charts](https://gitlab.com/ananace/charts).

### Performing the Helm Install

First, browse through the example values and the templates if required.
I managed to build a very minimal Matrix configuration via:

```bash
export KUBECONFIG=./kubeconfig.yaml 
helm repo add ananace-charts https://ananace.gitlab.io/charts
helm install matrix-synapse ananace-charts/matrix-synapse \
  --set serverName=example.com \
  --set publicServerName=matrix.example.com \
  --set wellknown.enabled=true
```

Or use a values file:

```yaml
serverName: example.com
publicServerName: matrix.example.com
wellknown:
  enabled: true
ingress:
  enabled: true
  tls:
    - secretName: matrix-synapse-crt
      hosts:
        - example.com
        - matrix.example.com
  hosts:
    - matrix.example.com
  className:
  annotations:
    kubernetes.io/ingress.class: "public"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
  includeServerName: true
```

Also **you should change the default postgresql credentials**:

```yaml
postgresql:
  enabled: true
  auth:
    password: <db-password>
    username: <db-username>
    database: <db-name>
```

Same things goes for the redis server:

```yaml
redis:
  enabled: true
  password: <new-redis-password>
```

### Configuring Upload and File Size Limits

```yaml
extraConfig:
  max_upload_size: 20M
ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "20m"
    nginx.ingress.kubernetes.io/client-max-body-size: "20m"
```

### Creating First User

To be able to login to your Matrix server you'll have to actually register a first user.

##### 1. Retrieve the Main Matrix Pod

```bash
export KUBECONFIG=./kubeconfig.yaml
export NS=<matrix-namespace>
export POD_NAME=$(kubectl get pods --namespace "$NS" \
  -l "app.kubernetes.io/name=matrix-synapse,app.kubernetes.io/instance=matrix-synapse,app.kubernetes.io/component=synapse" \
  -o jsonpath="{.items[0].metadata.name}")
```

##### 2. Run the 'register_new_matrix_user' command

```bash
kubectl exec --namespace "$NS" "$POD_NAME" -- register_new_matrix_user \
  -c /synapse/config/homeserver.yaml \
  -c /synapse/config/conf.d/secrets.yaml \
  -u "<your-user-name>" \
  -p "<your-strong-admin-password>" \
  --admin http://localhost:8008
```

### Setting up Element as Matrix Web-UI

Now you have a Matrix server, [you may connect to it with any client](https://doc.matrix.tu-dresden.de/clients/).
But you might also want to be able to use it directly in the web.

Element Web is the official web client for Matrix and provides a user-friendly interface for accessing your Matrix server through a browser.

To deploy Element Web, you'll need to create a values file that configures the default server URL and ingress settings:

```bash
export KUBECONFIG=./kubeconfig.yaml
helm upgrade --install element-matrix ananace-charts/element-web \
  -f values_element_web.yaml
```

Here's an example `values_element_web.yaml` configuration:

```yaml
defaultServer:
  url: 'https://matrix.example.com'
  name: 'example.com'
ingress:
  enabled: true
  tls:
    - secretName: element-matrix-crt
      hosts:
        - element.example.com
  hosts:
    - element.example.com
  className: "public"
  annotations:
    kubernetes.io/ingress.class: "public"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
```

Make sure to replace `example.com` with your actual domain and adjust the `defaultServer.url` to point to your Matrix server's public URL.

### Backup Existing Matrix Server

```bash
export NS=default
export KUBECONFIG=./kubeconfig.yaml

# List PVCs
kubectl get pvc -n $NS

# Find PostgreSQL secret
kubectl get secret -n $NS -o name | grep -i postgres

# Get PostgreSQL pod name
export PG_POD=$(kubectl get pods -n $NS -l app.kubernetes.io/name=postgresql -o jsonpath='{.items[0].metadata.name}')

# Backup database (replace <db-password> with actual password from secret)
kubectl exec -n $NS "$PG_POD" -- sh -lc '
  export PGPASSWORD="<db-password>"
  /opt/bitnami/postgresql/bin/pg_dump -Fc -h 127.0.0.1 -U "<db-user>" "<db-name>"
' > postgres-FULL-$(date +%F).dump
```

### Backup All Matrix Media

```bash
export KUBECONFIG=./kubeconfig.yaml   # adjust path if needed
export NS=default
export RELEASE=matrix-synapse

# Get one synapse pod name
export SYNAPSE_POD=$(kubectl get pods -n $NS -l app.kubernetes.io/name=$RELEASE -o jsonpath='{.items[0].metadata.name}')

# Create tar.gz backup of media directory
kubectl exec -n $NS "$SYNAPSE_POD" -- sh -lc '
  set -e
  cd /synapse/data
  tar czf /tmp/matrix-media-backup.tar.gz media
  echo "Done."
'

# Copy backup from pod to local machine
kubectl cp $NS/$SYNAPSE_POD:/tmp/matrix-media-backup.tar.gz ./matrix-media-backup-$(date +%F).tar.gz
```

### Use Matrix bot to automate messaging

**1. Login a Matrix user**

```bash
curl -X POST "<YOUR-MATRIX-SERVER>/_matrix/client/v3/login" \
  -H "Content-Type: application/json" \
  -d "{
    \"type\": \"m.login.password\",
    \"identifier\": { \"type\": \"m.id.user\", \"user\": \"<user-name>\" },
    \"password\": \"<your-strong-admin-password>\"
  }" -o login-response.json -w "\nHTTP Status: %{http_code}\n" -s -S
```

Use the token to (2) list registered users (3) send a message to a registered user.


Secure Federated Chat: Self Host Matrix on Kubernetes


[Open-Chat](https://github.com/msgmate-io/open-chat-go) is not in it's 7th iteration and fully re-written in golang.
It now as 'federation' features using libp2p, it allows you to quickly spin-up your own network of open-chat nodes.

Open-Chat network member can freely communicate with another, they are automatically joined into a completely independent self-healing overlay-network.

### Creating a Open-Chat Network

Open-Chat Networks can be established inside private networks, across public networks *or across both*.
To be able to penetrate across public and private networks open-chat need to be able to route traffic back to itself trough a router, 
or it need to have at-least one public relay peer available to use as alternate path.

#### **1. Prepare a Bootstrap and Relay Node**
On our public node, we want to use as 'relay' and 'bootstrap' peer, download the Open-Chat binary [from the github release page](https://github.com/msgmate-io/open-chat-go/releases)
Be sure to download the one with '-federation' build in.
E.g.: 

```bash
wget -O open-chat https://github.com/msgmate-io/open-chat-go/releases/download/build-10/open-chat-federation-0.0.111-linux-amd64
chmod +x open-chat # make it executable
```

#### **2. Start first federation node**

For the 'bootstrapper' node you'll have to specify base admin credentials and network-credentials
- network credentials allow 'read-only' network operations and are required to be known for joining a network
- admin credentials are individual to each node, they are required to perform any elevated privilege action, e.g.: creating proxy routes

All can simply be specified like this:

```bash
./open-chat --default-network-credentials "<network-name>:<password>" --root-credentials "<user>:<password>"
```

You can also initalize not credentials without revealing them by specifying `--root-credentials "<username>:hashed_<pdf-hashed-password>"`.
Now that our bootstrapper node is available we can use its peer-id to bootstrap other nodes.

> If you want to persis your open-chat configuration system wide just run it with `--install`

We can extract it's peer Id by either loggin in to the UI with the admin user, or via the Open-Chat cli:

```bash
./open-chat client login                # you'll be promted to ender username and password
./open-chat client identity -b64        # revals the node id in a re-usable base64 format
eyJpZCI6ICI8TE9DQUxfUEVFUl9JRD4iLCAiY29ubmVjdF9tdWx0aWFkcmVzcyI6IFsiL2lwNC88U09NRV9JUF9BRFJFU1M+L3RjcC8wL3AycC88TE9DQUxfUEVFUl9JRD4iXX0=
```

#### **3. Adding other nodes to the network**

The only thing you need is the bootstrap peer string and the networks credentials

```bash
./open-chat -dnc "<network-name>:<password>" -rc "<different-node-specific-admin-creds>" -bs "<base64-bootstrap-peer-string>"
```

Now we just need to wait a moment untill these nodes found each other and established connections.
Lets see if the nodes are connected.

```bash
./open-chat client login          # ....
./open-chat client nodes -ls    # list all available nodes
```

Now lets *see if we can send requests to the node*:

```bash
./open-chat client get-metrics --node <remote-peer-id>
```

Hurray it works! **Now where do we go from here?**

#### **4. Establish on-demand proxy routing rules**

When we have a network, we can model its traffic in any-way we like.
Open-chat nodes can configure arbitrary routing rules on per-node basis, 
every node can only route traffic that goes trough itself.

To route traffic trough another node, routing rules on that node need to be established, 
thus the admin credentials for that node must be known.

Since we control the full network we know admin credentials of very node and can establish any routing rules we want.

**Lets use open-chat to create a ngrok-like TLS to behind NAT reverse-proxy**, to achieve this we need:
1. A single public peer in our control with an `<PUBLIC-IP>`
2. A domain name A-Record pointing that IP `<HOSTNAME>`
3. A private behind NAT node that hosts some TCP service on `<peer-port>`

> All the steps presented here can also be run from a single node, by using network-requests that are build in to Open-Chat.
> For simplicity we run command on individual nodes, but there are also workflows that allow controlling routing rule requests to ANY node just from a single network member.

##### 1. Use open-chat to request TLS certificate on the public node

```bash
./open-chat client tls --hostname "<HOSTNAME>" --key-prefix "<self-defined-certificate-prefix>"
```
This will automatically start the ACME challenge process and saves the obtained certificate in the Open-Chat nodes database.

##### 2. Establish TLS to network routing rules

First we need to tell the public node to accept TLS traffic using the just obtained certificate.
It should only accept traffic coming from `"<HOSTNAME>"` and the traffic should be routed to a specific port on the system.

```bash
./open-chat client proxy --direction "egress" --traffic-origin "<HOSTNAME>" --traffic-target "<key-prefix>:<peer-port>" --kind "tls" --key-prefix "<self-defined-certificate-prefix>"
```

This create an 'egress' proxy that accepts incoming traffic from the `"<HOSTNAME>"` and runnels it into the network.

```bash
./open-chat client proxy --direction "egress" --traffic-origin "<own-peer-id>:<peer-port>" --traffic-target "<remote-peer-id>:<peer-port>" --kind tcp
```

This establishes an 'ingress' rule from the network to the remote peer node.
Now the only thing that is left to do is the local node to tunnel network traffic from that peer to the `<peer-port>` that runs the TCP service.

##### 3. Accept Network Traffic on a local Port

Traffic is now routes from the domain to the public peer, from there inside the network and to the remote peer.
But currently the remote peer would just drop the traffic as it doesn't know what to do with it, lets route it to the port that runs our service:

```bash
./open-chat client proxy --direction "ingress" --traffic-origin "<own-peer-id>:<peer-port>" --traffic-target "<remote-peer-id>:<peer-port>" --kind tcp
```


### Conclusion and Outlook

**This Article merely gives a hint what Open-Chat Federation is able to do**, options from there go way further.
Just as food for though, the following thins are only a few lines of open-chat rules away:

1. **Decentralized Multi-Site behind NAT hosting**
Host a swarm of Server behind NAT on different locations, route them trough multiple public peers in a 'load balancing' fashion
and you have a load resilient decentralized server infrastructure that masks the actual server locations, allows for easy traffic filtering and allows for easy horizontal scaling by just extending the node pool.
2. **Persistent self hosted Ngrok**. As shown in this article open-chat can easily be used to create development routing rules as tools like ngrok can do.
3. **Self managed overlay networks for anything**. This connect extends to any application, also the ones fully privately easily use open-chat routing rules to manage traffic in your local network, *you may even use it in offline local networks!*.


How Open-Chats Federation Enables anybody to host anything anywhere

Breaking Trough NAT: Decentralized Self-Hosting Via Open-Chat


N8n is a cool workflow and automation tool, that has a fair free plan and public code policy.
The community edition can be used and self-hosted by anybody, though it doesn't offer all the features n8n-cloud does.

For example the community edition doesn't allow synchronizing workflows and changes to a github repository.
But that's no issue really cause we can just craft a simple n8n workflow, that easily synchronizes all workflows into a private repo.

> If you don't have n8n set up yet, see my post on [self-hosting n8n on Kubernetes](/blog/n8n-on-kubernetes).

### Workflow setup

N8n has a n8n-trigger, for when an active workflow was updated.
This trigger can be used to run a workflow in the moment when it is saved.

Using this we can craft a simple workflow that accepts a n8n workflow ID and uses that and some github credentials to store the workflow as JSON inside a repository.

![N8n workflow](/static/assets/n8n_self_backup_workflow.png)

### (optional) Automatic Credentials Backup

Also if you followed my other blog post on [installing n8n in kubernetes](/blog/n8n-on-kubernetes), you can also add a simple workflow that allows you to automatically backup your n8n credentials, every time a workflow is changed.

This workflow runs a GitHub Actions workflow and waits for its completion. The GitHub workflow uses stored secrets and an encrypted data store to access your Kubernetes cluster, downloads the credentials from the running n8n pod, then saves them encrypted to your repository.

The setup handles all the Kubernetes complexity—you just need to configure GitHub secrets for cluster access and point the workflow at your n8n namespace.

> If someone wants the actual workflow configurations, they should write me an email at `tim+blog@timschupp.de`.

A n8n workflow setup, that synchronizes workflow and credential changes directly to github

n8n community edition: synchronize workflows to github


n8n is a flexible, open-source workflow automation tool that runs well on a small self-hosted cluster. Below is a lean, production-minded setup using a maintained Helm chart. This mirrors my usual pattern: keep it simple, make it reproducible, and put it behind ingress with TLS.

### Helm chart

Maintained chart: [`8gears/n8n-helm-chart`](https://github.com/8gears/n8n-helm-chart) — solid defaults, clear values, and active maintenance by the 8gears team. Kudos to the authors and maintainers for keeping queue mode, probes, lifecycle hooks, and ingress settings tidy and well-documented.

> If you’re new to Kubernetes on a VPS, see my Microk8s setup first: [/blog/microk8s-on-vps](/blog/microk8s-on-vps)

### Example values

Sanitize hosts and secrets for your environment. This example uses `ClusterIP` with `nginx` and `cert-manager`.

```yaml
persistence:
  enabled: true

image:
  repository: n8nio/n8n
  pullPolicy: IfNotPresent
ingress:
  enabled: true
  className: "nginx"
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  hosts:
    - host: n8n.t1m.me
      paths:
        - "/"
  tls:
    - hosts:
        - n8n.t1m.me
      secretName: n8n-example-com-tls
main:
  replicaCount: 1
  deploymentStrategy:
    type: Recreate
  persistence:
    enabled: true
    type: "dynamic"
    mountPath: "/home/node/.n8n"
    storageClass: "<your-default-storage-class>"
    size: 8Gi
    accessModes:
      - ReadWriteOnce
    annotations:
      helm.sh/resource-policy: keep
  nodes:
    builtin:
      enabled: true
      modules:
        - ws
      reinstallMissingPackages: true
  service:
    type: ClusterIP
    port: 5678
  config:
    n8n:
      host: "n8s.<your-domain>"
      protocol: "https"
      port: 5678
      editor_base_url: "https://n8s.<your-domain>"
      webhook_url: "https://n8s.<your-domain>"
      secure_cookie: "true"
      proxy_hops: 1
      log_level: "info"
      diagnostics_enabled: "false"
    db:
      type: "sqlite"
  secret:
    n8n:
      encryption_key: "CENSORED"
scaling:
  enabled: false
  worker:
    replicaCount: 0
  webhook:
    enabled: false
    replicaCount: 0
```

Notes:
- For quick single-node or hobby deployments, `sqlite` is fine. For multi-node or longevity, pick Postgres.
- Set `encryption_key` to a long, random value. Rotating it will invalidate existing encrypted credentials.
- If you want queue mode, enable `scaling.enabled` and configure Redis (internal `valkey` or external Redis) as per the chart docs.
- The `persistence` block ensures your workflows and credentials survive pod restarts.
- `helm.sh/resource-policy: keep` prevents accidental data loss during Helm uninstalls.

### Install/upgrade commands

Run with your `kubeconfig.yaml` and a dedicated namespace:

```bash
KUBECONFIG=./kubeconfig.yaml kubectl create ns n8n
KUBECONFIG=./kubeconfig.yaml helm uninstall n8n -n n8n
KUBECONFIG=./kubeconfig.yaml helm upgrade --install n8n oci://8gears.container-registry.com/library/n8n --namespace n8n -f n8n-values.yaml
```

After the release becomes `READY`, n8n should be reachable at `https://n8n.t1m.me` (or your configured host).

### Verify persistence before going live

**Critical**: Before creating workflows or adding credentials, verify that the persistent volume is working correctly. A failed volume mount means data loss on pod restarts.

Check the pod status and volume mount:

```bash
KUBECONFIG=./kubeconfig.yaml kubectl get pods -n n8n
KUBECONFIG=./kubeconfig.yaml kubectl describe pod -n n8n -l app.kubernetes.io/name=n8n
```

Look for:
- Pod status: `Running`
- Volume mount: `/home/node/.n8n` should be mounted from your PVC
- No volume-related errors in events

Test persistence by creating a simple workflow, restarting the pod, and verifying it still exists:

```bash
# Restart the pod to test persistence
KUBECONFIG=./kubeconfig.yaml kubectl delete pod -n n8n -l app.kubernetes.io/name=n8n
```

If your workflows disappear after the restart, the volume mount failed. Check your storage class and PVC status:

```bash
KUBECONFIG=./kubeconfig.yaml kubectl get pvc -n n8n
KUBECONFIG=./kubeconfig.yaml kubectl describe pvc -n n8n
```

### Credential management via CLI

n8n includes a CLI for bulk credential operations. Access the container and use these commands:

```bash
# Get into the running pod
KUBECONFIG=./kubeconfig.yaml kubectl exec -it -n n8n deployment/n8n -- /bin/sh

# Export all credentials (decrypted)
n8n export:credentials --all --decrypted --output=credentials.json

# Import credentials from file
n8n import:credentials --input=credentials.json
```

This is useful for:
- Backing up credentials before major updates
- Migrating between n8n instances
- Disaster recovery scenarios

**Security note**: The `--decrypted` flag exports credentials in plain text. Handle these files with extreme care and delete them immediately after use.

### Bulk restore / import n8n json workflows

For migrating workflows from another n8n instance or restoring from backups, you can bulk import JSON workflow files directly into your Kubernetes deployment.

Download and run the restoration script:

```bash
curl -fsSL https://raw.githubusercontent.com/tbscode/tims-blog-posts/refs/heads/main/assets/restore_workflow_json_files.sh -o restore_workflow_json_files.sh
chmod +x restore_workflow_json_files.sh
export KUBECONFIG=<your-kubeconfig>
SRC_DIR=<local-dir-containing-workflows>
./restore_workflow_json_files.sh $SRC_DIR
```

The script will:
- Find your running n8n pod automatically
- Upload all `.json` files from the specified directory
- Import each workflow using the n8n CLI
- Provide a summary of successful/failed imports
- Clean up temporary files

This makes it trivial to migrate workflows between n8n instances or restore from backups. The script handles all the Kubernetes complexity, so you just point it at a folder of JSON files.

> For automatic workflow and credential synchronization to GitHub, see my post on [synchronizing n8n workflows to GitHub](/blog/n8n-auto-saving-workflows-community-edition).

### Hardening checklist

- Switch to Postgres for production and enable persistent volumes.
- Keep `secure_cookie: true` and TLS on end-to-end; n8n stores credentials for integrations.
- Limit ingress access (e.g., IP allowlists) if you only need the editor from trusted networks.
- Set `N8N_USER_MANAGEMENT_DISABLED=false` and create dedicated users; enable 2FA on your IdP if applicable.
- Regularly backup your persistent volume and test restore procedures.
- Monitor PVC usage to avoid running out of storage space.

### About the chart authors

The chart is maintained by the 8gears team and contributors. The repository is here: [`8gears/n8n-helm-chart`](https://github.com/8gears/n8n-helm-chart). They’ve done a great job exposing n8n’s core options (readiness/liveness probes, deployment strategies, webhook and worker modes, and Redis/Valkey integration) in a clean values structure.

That’s it—small, clean, and easy to back up. Perfect fit for a self-hosted automation hub.

### Installing community nodes

Enable them once and you’re done:

1) Set the env var so the UI shows the Community Nodes menu: `N8N_ENABLE_COMMUNITY_NODES=true` (in the chart, add to `extraEnv`).  
2) Keep a short allowlist if you want: `N8N_COMMUNITY_PACKAGES_ALLOWLIST=["n8n-nodes-<package>"]`.  
3) Restart the pod, then in n8n go to *Settings → Community nodes → Install*. Paste the package name (e.g., `n8n-nodes-openai`), install, and reload the canvas.  
4) If your cluster egress is locked down, allow outbound HTTPS to `registry.npmjs.org` so installs don’t hang.


Minimal, repeatable setup to run n8n with TLS behind nginx ingress using the 8gears Helm chart.

Self-Hosted n8n on Kubernetes (Helm chart)


A quick, repeatable way to run WordPress on Kubernetes using the maintained TrueCharts chart. Small footprint, TLS via cert-manager, and a clean ingress.

### Chart

The TrueCharts WordPress chart ships via OCI and builds on the TrueCharts common library for consistent values and sane defaults.

- Chart page: [truecharts.org/charts/stable/wordpress](https://truecharts.org/charts/stable/wordpress/)

```bash
helm install wp oci://oci.trueforge.org/truecharts/wordpress \
  --namespace blogging \
  --create-namespace \
  -f values.wordpress.yaml
```

### Example values

Sanitize domains, credentials, and secret names for the target setup.

```yaml
ingress:
  main:
    enabled: true
    ingressClassName: public
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      nginx.ingress.kubernetes.io/proxy-body-size: "0"
    hosts:
      - host: wp.<your-domain>
        paths:
          - path: /
            pathType: Prefix
    tls:
      - hosts: [wp.<your-domain>]
        secretName: wp-<your-domain>-tls

wordpress:
  user: "<admin-username>"
  pass: "<strong-password>"

workload:
  main:
    replicas: 1
    podSpec:
      containers:
        main:
          env:
            WORDPRESS_EXTRA_WP_CONFIG_CONTENT: |
              define('WP_HOME','https://wp.<your-domain>');
              define('WP_SITEURL','https://wp.<your-domain>');

resources:
  requests:
    cpu: "500m"
    memory: "1Gi"
  limits:
    cpu: "4"
    memory: "8Gi"
```

Notes:

- Ensure `ingress` and `cert-manager` are installed and a `ClusterIssuer` named `letsencrypt-prod` exists.
- Set DNS for `wp.<your-domain>` to the ingress IP. After install, visit `https://wp.<your-domain>`.
- TrueCharts maintains a broad catalog and a shared common chart, providing consistent configuration across apps; the WordPress chart is kept up-to-date and is straightforward to automate.




Set up WordPress on a self-hosted Kubernetes cluster using the TrueCharts Helm chart.

WordPress on Kubernetes with TrueCharts (short guide)


Uptime-Kuma is a lightweight status and monitoring dashboard suited for homelabs or lean production clusters. Below is a minimal, repeatable setup to run it behind an ingress with TLS.

### Helm Chart

A maintained chart is available: [dirsigler/uptime-kuma-helm](https://github.com/dirsigler/uptime-kuma-helm), from [`@dirsigler`](https://github.com/dirsigler/uptime-kuma-helm).

```bash
helm repo add uptime-kuma https://helm.irsigler.cloud
helm repo update
helm upgrade my-uptime-kuma uptime-kuma/uptime-kuma \
  --install \
  --namespace monitoring \
  --create-namespace \
  -f values.uptime-kuma.yaml
```

### Preview

The following screenshot shows how Uptime-Kuma presents monitoring data for this blog (`blog.t1m.me`).

![Uptime-Kuma dashboard for blog.t1m.me](/static/assets/uptime-kuma-dashboard.png)

### Example values

Sanitize hosts/secrets for the target environment. This example runs as a `ClusterIP` behind `nginx` and cert-manager.

```yaml
ingress:
  enabled: true
  ingressClassName: public
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: public
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://<your-domain>"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, OPTIONS"
    nginx.ingress.kubernetes.io/cors-allow-headers: "Content-Type"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "false"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/server-snippets: |
      proxy_buffering off;
      proxy_cache off;
  hosts:
    - host: status.<your-domain>
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: uptime-kuma-tls
      hosts:
        - status.<your-domain>

persistence:
  enabled: true
  size: 5Gi

service:
  type: ClusterIP
  port: 3001

env:
  TZ: Europe/Berlin
```

A helper script is available for quick install and values generation:

- Download: [`setup_uptime_kuma.sh`](https://github.com/tbscode/tims-blog-posts/blob/main/assets/setup_uptime_kuma.sh)
- Make executable and run with env vars:

```bash
curl -fsSL https://raw.githubusercontent.com/tbscode/tims-blog-posts/refs/heads/main/assets/setup_uptime_kuma.sh -o setup_uptime_kuma.sh
chmod +x setup_uptime_kuma.sh
KUBECONFIG=./kubeconfig.yaml HOST="status.<your-domain>" ./setup_uptime_kuma.sh
```

Notes:

- Keep persistence on; monitors would otherwise need reconfiguration after pod restarts.
- On Microk8s, ensure `ingress` and `cert-manager` are enabled and a `ClusterIssuer` named `letsencrypt-prod` exists.
- The app becomes reachable at `https://status.<your-domain>` after DNS is set.

That’s it—small, stable, and easy to back up. Perfect for keeping a quiet eye on your stack.


### Alerts and hardening

- Enable certificate expiration notifications. Even with cert-manager in place (see the Kubernetes setup post), missed renewals often indicate underlying cluster health issues.
- Configure Twilio SMS alerts to receive notifications when an endpoint goes down and when it recovers. The following screenshot shows a brief incident where a page returned 503 and then back to 200 (minimal downtime during a controlled server migration):

![Incident timeline 503→200](/static/assets/kuma-uptime-incident-503-200.png)

- Enable 2FA for the admin account.

![Uptime-Kuma 2FA](/static/assets/2fa-uptime-kuma.png)

- Set up automated backups. Creating a daily full copy of `/app/data` and keeping a sensible retention schedule is recommended.




Tiny guide to deploy Uptime-Kuma on a self-hosted Kubernetes cluster using a maintained Helm chart.

Uptime Monitoring on Kubernetes by self-hosting Uptime-Kuma



In light of Twilio Video Calls reaching its end of life, the quest for a new video call service provider - for our startup [Little World](https://little-world.com) - is essential for seamless communication experiences. This comparison focuses on technical capabilities, overlooking aspects such as server hosting locations and GDPR compliance which are beyond the scope of this review.

I've experimented & compiled the following table, to compare various services based on criteria essential for React-based web and native applications. These criteria include unique authentication for rooms, server-side token generation, webhooks, device detection and permissions, compatibility across platforms, mobile redirection capabilities, pricing, React SDK availability, self-hosting possibilities, and practical implementation tests.

| Feature / Service | [LiveKit](https://docs.livekit.io/realtime/) | [Dyte.io](https://dyte.io/) | [Jitsi Meet Self Hosted](https://jitsi.org/) | [Jitsi 8x8 Conferencing](https://8x8.vc/) | [Zoom](https://zoom.us/) |
|-------------------|:--------------------------------------------:|:-------------------------:|:-------------------------------------------:|:-----------------------------------------:|:-----------------------:|
| Unique Authentication Required Rooms | Yes | Yes | Yes | Yes | Yes |
| Server-side Token Generation | Yes | Yes | Yes | Yes | Yes |
| Webhooks (Join/Leave) | Yes | Yes | [No #1](#1) | Yes | Yes |
| Built-in Device Detection & Permissions | Yes | Yes | Yes | Yes | Yes |
| Cross-Platform Compatibility | [Yes (React Native) #2](#2) | [Partially (No WebView) #3](#3) | [Yes #5](#5) | [Yes #5](#5) | [Partially #5](#4) |
| Mobile Redirect for App Calls | No | [Uncertain #5](#5) | Yes | Yes | Yes |
| Affordable & Scalable Pricing | Yes | Yes | Not really (Consider Maintenance) | [No #6](#6) | [Yes #7](#7) |
| React SDK Availability | Yes | Yes | Yes | Yes | [No #8](#8) |
| Self-Hosting Capability | Yes | No | Yes | No | No |
| Practical Implementation Test | Not yet | No | Yes but no sockets! | Yes | Yes |

###### Notes:

- <a id="1"></a>**#1 Jitsi Meet Self Hosted**: Webhooks would require custom implementation.
- <a id="2"></a>**#2 LiveKit**: Exclusive React Native integrations for Android and iOS.
- <a id="3"></a>**#3 Dyte.io**: Limited to native apps, lacks in-browser video call support.
- <a id="4"></a>**#4 Zoom**: Compatibility issues noted with WebView; [native or react-native Zoom clients necessary](https://developers.zoom.us/docs/video-sdk/react-native/)
- <a id="5"></a>**#5 Jitsi**: Even has mobile webview / capacitor support ( no need for native / react native )
- <a id="6"></a>**#6 Jitsi 8x8 Conferencing**: Expensive for broader usage.
- <a id="7"></a>**#7 Zoom**: Pricing model based on usage; but specificly low entry pricing / many free minutes per month
- <a id="8"></a>**#8 Zoom**: The SDK constrains customization options, offering default Zoom UI.

For developers seeking to replace Twilio Video Calls, this comparison sheds light on several viable alternatives. Each service offers a unique blend of features and constraints, making it crucial to align the choice with the specific needs of your application and user base. 

Further exploration and prototypes are valuable steps toward identifying the most suitable video call service. Additionally, reviewing discussions and feedback, such as those found in [community forums](https://www.reddit.com/r/node/comments/18bain4/list_of_top_twilio_video_alternatives/), can provide insights and considerations from a broader development community.

#### Conclusion

To me, it seems that LiveKit is the most promising solution. It allows full UI customization, has the required socket integrations, React Native SDKs, and can even be self-hosted. I was surprised to see how difficult and annoying it is to use the Zoom SDK with a custom video call UI; other than that, it's probably the best in terms of pricing. Jitsi is also quite good in this regard, but the missing WebSocket integrations (offered by 8x8 Conferencing) make it hard to use for web applications.

Dyte also looks fairly strong, but I haven't investigated it as much. In my opinion, it loses to LiveKit for not being open-source or self-hostable.

This comparison is not exhaustive in any regard, please contact me for comments or improvements!


#### Additional Options To Consider:

- https://docs.videosdk.live/
- https://www.daily.co/


A comprehensive comparison of leading video call service providers focusing on cross-platform compatibility for React-based web and native applications.

Comparison of Video Call Services for Cross-Platform React Web/Native Apps


### Full Setup Lenovo Yoga 9 Pro Regolith Linux (Ubuntu 23.10 + i3)

This is my setup guide for Linux on my new laptop. I'm using the newest 23.10 Ubuntu because I want to.
This blog intends to report how I got things set up & the open issues I still have.

> Don't hesitate to contact me if you have possible solutions or questions regarding my setup, cheers :)

This is the labtop in question: https://geizhals.de/lenovo-yoga-pro-7-14aph8-storm-grey-82y80020ge-a3000891.html?hloc=at&hloc=de

- 14.5", 2560x1600 (WQXGA), 16:10, 208ppi, 90Hz, non-glare (matt), IPS, 350cd/​m², Blaulichtfilter, Dolby Vision, 1.500:1, 100% sRGB 
- AMD Ryzen 7 7840HS, 8C/16T, 3.80-5.10GHz, 16MB+8MB Cache, 35W TDP, 54W cTDP, Codename "Phoenix" (Zen 4, TSMC 4nm) 
- AMD Radeon 780M (iGPU), 12CU/768SP, 2.70GHz, Architektur "RDNA 3" (Phoenix) ( no additional GPU )
- 32GB LPDDR5X-6400 (32GB verlötet, nicht erweiterbar) 


## Open Issues:

1. Keyboard backlight levels not working correctly
It goes from bright to medium to bright to out. But medium is also very bright.
2. Master sound level not working (generally sound seems off)
I've also played with a lot of different output options in `pavucontrol`,
some seem to be completely broken and sound super metallic, while others sound pretty good at full volume but then start sounding weird when leveling down the audio.
This is significantly annoying but ok since it can be worked around by leveling the specific output sound level instead of using the volume buttons.

## Initial setup

The laptop comes with Windows originally. So I:

- create a boot USB with Ubuntu 23.10.
- disable BitLocker in PowerShell
- disable hibernation in PowerShell
- in Windows prepare an unformatted partition of the hard drive
- in BIOS disable secure boot
- boot & install Ubuntu 23.10
- install regolith-desktop using package: https://regolith-desktop.com/

## Configuring Regolith

Basic Regolith / i3 looks can be configured in `~/.config/regolith3/Xresources`.
E.g.: I like to adjust gap and border sizes & colors:

```
wm.gaps.inner.size: 1
wm.window.border.size: 3
wm.client.focused.color.child_border: #AAD3E9
```

## Resolution

- Falsely detected out of the box required xrandr adjustments.
- issues with fractional scaling if not on Ubuntu desktop (so i3)

I found that the i3 window header and bar sizes seem to depend on the dpi set in `~/.Xresources`.
`Xft.dpi: 100`. Weirdly, a higher value causes bigger i3 bar and UI and a lower one smaller.

> I found the cursor size set here to have no effect `Xcursor.size: 5`

To view the current resolution config `xdpyinfo | grep -B 2 resolution`:

```
screen #0:
  dimensions:    2560x1600 pixels (677x423 millimeters)
  resolution:    96x96 dots per inch
```

## External Monitors (& USB monitors)

At first, I thought the external monitors were not working but I soon realized that they were detected but not set to `active`, simply going into `arandr`, setting them to active, and then applying the configuration made all external monitors work as expected.

### Persisting xrandr settings using autorandr

`sudo apt install autorandr` just set the correct settings using `xrandr` or a UI like `arandr`,
then save the profile as default `autorandr --save default`

### Gnome Scaling Factor

I wasn't yet able to get fractional scaling working on Ubuntu yet.
But I found that the overall scale can be set from the command line using:

```bash
gsettings set org.gnome.settings-daemon.plugins.xsettings overrides  "[{'Gdk/WindowScalingFactor', <1>}]"
gsettings set org.gnome.desktop.interface scaling-factor 1
```

It accepts scaling factors `1`, `2`, `3`

## Setting up Keepmenu as password manager

I love using Keepmenu to easily manage a local password database.

- https://github.com/firecat53/keepmenu

I use it with `rofi`; this is my config: `~/.config/keepmenu/config.ini`:

```ini
[dmenu]
dmenu_command = rofi -show drun -dpi 1

[dmenu_passphrase]
nf = #222222
nb = #222222
rofi_obscure = True

[database]
database_1 = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
keyfile_1 = 
pw_cache_period_min = 30
autotype_default = {USERNAME}{TAB}{PASSWORD}{ENTER}
```

### Setting up the keybinding

The easiest way is to just use the GNOME system keybindings like described in: https://regolith-desktop.com/docs/using-regolith/configuration/

Just open the GNOME settings `Super+c` & open the keybindings editor in the 'Keyboard' settings.

You could also set it up using an i3 config e.g.:

```bash
bindsym $mod+Shift+K exec "DO SOMETHING"
```

## Setup screenshot command

`sudo apt install gnome-screenshot`

I like to map it to `Super+Shift+U`

Then setup a keyboard shortcut like this: `gnome-screenshot -acf /tmp/test && cat /tmp/test | xclip -i -selection clipboard -target image/png`
[as per this stack overflow answer](https://askubuntu.com/questions/1196914/gnome-screenshot-cant-copy-to-clipboard-in-ubuntu-18-04).

I saw only this stategy working, copying directly with `-acf` wan't working for me.


A comprehensive setup guide for setting up a Lenovo Yoga 9 Pro with Ubuntu 23.10 and Regolith i3, including resolution and external monitors configuration, and workarounds for common issues.

Full Setup Guide: Lenovo Yoga 9 Pro with Regolith Linux (Ubuntu 23.10 + i3)


- [the python script](https://github.com/tbscode/tims-blog-posts/blob/main/assets/send_slack_message.py)

Need to notify your team about deployment status on Slack? Automate the process using the slack_sdk Python package with GitHub Actions. Here's how to set it up step by step:

### Creating Your Slack App for Notifications

1. **Create a Slack App**: Go to the [Slack API](https://api.slack.com/apps) section, select 'Create New App', input a name, and choose your workspace.

2. **Set Permissions**: In the Slack app settings, navigate to 'Features' > 'OAuth & Permissions'. Under 'Scopes', add `chat:write` for your bot to send messages. Once you save changes, click 'Install to Workspace'.

3. **Retrieve Your Slack Bot Token**: After installing the app to your workspace, grab your 'Bot User OAuth Token' from the 'OAuth & Permissions' page. Securely store this token as it will be used to authenticate your GitHub Actions script.

### Setting Up GitHub Actions to Send Slack Notifications

First, include your Slack API token as a secret in your GitHub repository:

1. In your GitHub repo, go to 'Settings', then 'Secrets'.
2. Select 'New Repository Secret', and add your Slack API token under `SLACK_API_TOKEN`.

Now, integrate the `send_slack_message.py` script with your deployment process:

```python
# send_slack_message.py
import os
from slack_sdk import WebClient

client = WebClient(token=os.environ["SLACK_API_TOKEN"])

response = client.chat_postMessage(
    channel=os.environ["CHANNEL_ID"],
    mrkdwn=True,
    text=os.environ["MESSAGE"],
)

print(response["message"]["text"])
```

Add a GitHub Actions step in your workflow to execute the notification script:

```yaml
jobs:
  deployment:
    runs-on: ubuntu-latest
    steps:
      # ... previous steps for deployment ...
      
      - name: Notify Slack Channel
        env:
          SLACK_API_TOKEN: ${{ secrets.SLACK_API_TOKEN }}
          CHANNEL_ID: 'your-channel-id'
          MESSAGE: "*Deployment Update*\nImage: ${{ steps.build.outputs.IMAGE_URL }}\nURL: ${{ steps.deploy.outputs.WEBSITE_URL }}"
        run: |
          pip install slack_sdk
          wget -qO- https://raw.githubusercontent.com/tbscode/tims-blog-posts/main/assets/send_slack_message.py | python3 -
```

Replace `your-channel-id` with the actual Slack channel ID where you wish to send notifications. 

> Replace the script url with a self hosted fork of [send_slack_message.py](https://github.com/tbscode/tims-blog-posts/blob/main/assets/send_slack_message.py)

Remember to update the `MESSAGE` variable with relevant details for your deployment, such as image URL or site URL.

Executing the GitHub Action with these configurations will automatically send your predefined message to the designated Slack channel, informing your team about the latest deployment status.

Your GitHub Actions workflow now works seamlessly with Slack for real-time notifications!

Set up Slack notifications for deployments using GitHub Actions and the Python slack_sdk package, keeping your team updated automatically.

Send Slack Notifications on Deployments with GitHub Actions


# Solid Authentication Overview

> A presentation prepared for the
> 'Data Ecosystems Lab'

## Content

This blog article delves into several key aspects of Solid authentication and OpenID Connect, providing insights into the mechanisms and practical implementation details:

- **Introduction to Solid and OpenID Connect**: An overview of the Solid framework and how OpenID Connect (OIDC) serves as a foundational technology for secure authentication.
- **The Solid Community Server's Authentication Process**: A closer look at how the Solid Community Server implements OIDC and the intricacies involved in user verification.
- **Reverse Engineering Authentication with Python**: Step-by-step guidance on how to reverse engineer client authentication processes using the Python `requests` library.
- **Jupyter Notebook Demonstration**: A practical example showcasing the authentication flow within a Jupyter notebook, accessible for interactive learning and experimentation.
- **Flow Charts and Visual Aids**: The inclusion of original and comprehensive flow charts to visualize the authentication sequence as defined by the Solid project.

> If you want to deploy solid on a self hosted kubernetes cluster [checkout this blog post](todo)

## Presentation Slides

<iframe src="https://docs.google.com/presentation/d/e/2PACX-1vR9IbBdC5t9iGfPpiIm6voYEIJvfLBghO4rQoI1J57_cqfim40nndh1kjCajXyoTt77bQVvMgLaLkDJ/embed?start=false&loop=true&delayms=5000" frameborder="0" width="960" height="569" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true"></iframe>

## Flow Chart

- To view the original flow chart as published by the Solid project, please visit this link: [Solid OIDC Flow Chart](https://solidproject.org/TR/oidc)
- For a full-size preview of the flow chart, click here: [Full Size Flow Chart Preview](https://viewer.diagrams.net/?tags=%7B%7D&highlight=0000ff&edit=_blank&layers=1&nav=1&title=diagramm.xml#R7V3bcptIGn6WuVDV7oUozodLW46SbM3UeOKkUnuVQqIlsUFqBpDszNNvNydB8yNAAoRkchHbTQMN%2Ff3nAxNptn376Jnu5g9sIWci8tbbRHqaiKKgqgb5QUd%2BRSOywUcDa8%2B24knHgRf7HxQPJtP2toX83MQAYyew3fzgEu92aBnkxkzPw6%2F5aSvs5O%2FqmmtUGHhZmk5x9LttBZtoVFf44%2FgnZK83yZ0FPj6yNZPJ8YC%2FMS38mhmSPkykmYdxEP22fZshh7685L1E581LjqYL89AuqHOC%2BYfh%2FTpIbwdNdP%2FGb389ffn0YRpf5WA6%2B%2FiBJ6LqkOs9rjC5LHlh5jI6oP69pyt9nOHtCnuBaR6H6PaagTmlp0x9b5mbvwkCulUPdGHinE7xuTXGaweZru1zS7wlw0ufTJmvzK3tUJyA91DX9Oc3H3nJEuPrJYfi1xz8SvaOvHGX%2FrrfOr%2FbK%2BTYO%2FLXo4s8e4sCchnpyYmHn49jj68bO0AvbvTcrwTV4UNsHfKXQH4lQAtMcgo9nSd%2FWx52v5reGgXxwBI7jun69iJcBh3x0HLv%2BfYBfUF%2BBHA6ivcBvfUsBS4ddMljkxE%2F8Mg96CAy%2FeAV%2BfT5duj1g0XAmTyg9nh8PSg9cHxj9J2nx50Fzpydea0zZlLJBQ7IC2xCFiWnkgc8ICt3Lp%2Bb4OH9zirM0J4yh6L3W4R1glGyAvSWGYph%2FhFhsnXeLzIlPipLnCBFJyVcJ6bB1yMJp4S6yZCvnhCrGbONdXrxI2WRX2LiakBoIkBoDFwzIHMx2f1wCcrjRHliUEswssFrvDOdLG5BPAUhMl8oIfhFSniHYCvC6yRfrI85JQc4DQJcEW%2Ba0hXelCrG3gPHrik8Iu79SGX1yNvvjNxa5%2B0iL3K6UMncxV6Zu3ZrxEbePyU1%2FjtafLbqklyGUBLEPDj2ekfGFjgIyErpha0HqnXTMQcvf%2BbgSxCfh2oCS%2BkphWAKt5i8Uvzwp%2FDj470XP%2F1J8RskxFw2T43m0TWfRKOHHDMgVJ%2B3NwBcxac%2BU4F%2BRLFg5MWFIDHQjB4oPuuITvJuzV%2BZabGeUHqfVOgk91EYG4Fdl8rMl0%2FPT2kPvj75JVoxc3ayfLxa%2BSiYsOSXvufzKVJvpG7VgTN20Y6KBNPfpPyMHI4tVv0KQN8j5eXpBf81fVlMZ%2Fajo60%2B4uTBuwawqDIAVhkAR5R2MYAlFpBqLYAlSplYscoSMjsDhac243bkQiwP%2BGcPH2zrqI0tvGTGw57IWJF%2FQQS7wGGiBUUILZ8yanh3rOHFb0g5ycQaWFWyxuVJGFL2JEjZ41tQ9kCiNhqJltGS78%2BSP8mDa2OOlThGPVNeEFoA3M7531ravL18M5T%2FPqFv%2F3w7vO6nYjNlZkRct4grIKkuCE8gjpPyJq0hJmyvAnWJ26l1LidADstB6y4e%2BntP8ZdatfwTXu63KFQx7t7AjQ3XrIF7kpn0biCwns6WLFxJOcsCLTMQ2FWWmDEdmbIwJd6cd%2BlIiX9%2BnpFTaHB2Za%2FPNwXujDBPqZWd06WWpxexK7pkPENihSdJUhRwXTXpmL3bEOm40htAbE%2Fs5YhsIkp8%2BC9Hi0fKSAYj4g9tL0LrvMC7bxF5NSXg%2FgL5Hz98nVCH%2FNxCWzx1MRHYc9fDK5tqffOl6ZEBaYuaegsIiZqOgxy89swto%2FvmjjVwGqzsN5Q4%2BqK%2FHWeW7pW00pdouaTGb%2BDhnyhzZKErclumsGjAgi6rIYpQTLsNuwQEtHiDgmlt%2B1HkY%2BbYGc1wFEkDEklSR8EQiYnRJ%2Fc5T1UsrHIAIqahp%2BCOwh5GEdLXjO%2BxTqRUG2o7PMJoTZLcCNISfHZf4ZGEgkaNKJQxz3%2B%2BxCoRh%2B0l1YeIvBr1nwTqYomUyKV9pOGCXjQgSaoCcBPtxN7ZgenQ10yAH6snvBnG2zrRSXY49OS2yL99wtYCVuUJB%2Be2kyyNrCD%2Ba1gpHTHOeI7n4yVcmuTBM4g1utFrROY%2BMl%2BR5CE2nK8p4HPUEzKiUvEWBqA3SVBMrz9BNDQzKX9XHzu2pXA%2BYSSmMyX35Sx0CA%2BNkimiH0HnGBKBYoZ6v7JJrpRN7wrTzLMP9ZmixXIcV0pdt%2B8vsxSkWzJElbq4kFT1QqpM%2FGPCAGhQbosGz4G675q7M6EeXSjYutO9701DCTCR%2Ba1pO1wgbLktPfMr8gNBlH7LIC6643lpXvcHxPjoVJBY8SDLxei%2BmJnWCzgVyPpuxaLox1mUMzZiN1XW1uDr2xqMbVDDqBDhLb%2FQYmCywuWaqnJTg0HQNPA%2BdRX6Qk5eTcOmLXVdaZaCdztuzu7dkuzW1Q2UNve1S%2BB96kFMYSzaukHq1vzorTkmh6MGV%2BsBRokakEZquVfy7qY%2Fd%2FiVSPk5JRnbmka5HXuP4BLvbkDo9xSt1QtCH0pd7ixgC%2Bd%2BVaH6ltCql6A1ahBAKxioaURYVL7IrXmkN5%2FgXgG%2BEpwAaCp3dKscE2rUgVzQEm2xjRJHEDuV3oQ7ws4HWh1JkfM7XtvUhJp5yPJvCEHF%2FgcqYAx3CKFTjvgxjX0QaeyXibeCDSCI9RLWRa2rjHV5LMy5Y3zJIsDBIOWJLfFsD18j%2F7obfCl8AV9gi6Be8VWZTtnECUwAsfO3Nj3NibUYqqF1lEoQxvq%2Fbuxdu46SnIsvdi6Wu%2FhOdhjp37Ui6zSVIPNPzV%2BzraxGNvofOxFLE%2B2ZlhCyXjHfYOc38%2BQUnYWMAjCA%2BL9y3Vjp0BLRyGTbcsn%2FMecYQ%2F3HVJsCnFVI7%2B01jlMZZOwyeFhmStM%2BEeQ9Ex4f0Hy2bNtG0IA%2BJ3R4hl19YV%2FB%2FN6nburs3iu97v1YUwQyrtAp7W1H1nWKdSlQhkRXrMvfLIld8pNoerP51zfnO7%2F7tAe7z16miA4yjpdPZgXfhNSRUlqIQiiSwWmMJto45leqLCp8rRt2HR%2BubH55DYn4Ja2KnkXcKY6Y8XNMnc5JQRr%2FsFwi%2F7S3%2BSaEpViEAuTD0cCUrDac0CChiW1CIxUz79SorizLSnyylaWGcm%2BFWUVU1mwK19gwl%2FKGs6JVVMDrOiugGzW7g3hvza58HdnaIAEOkjd%2F3tmBHdbeWGTWV6KF0ct9QZbtUUfszfNiRWBjyooG2K2i2FV6LAiFcoenZR9qb%2FHCXP5ch290Ghs7dKftaEcjAyQz7q0X%2F4o8zXzy49%2BxMUOhklgZD6GYTnB3PF4ja3aiPEq8RF3%2BSS2ftSQ%2Fo0IuYo%2FIE%2B0xKu%2F64cUA%2B7H3bOrbPo0qEH65N%2FXebBpB4TlVqsR1NqWin84ILX1Wo2UmZ1GHzIo8nmpu6XbuFj79kWqeX8L2CEkmF%2F9hZ7mRWLgpZifoeU4H9uKHvDRt6Jxw55cbasbfBryStk%2BffX9%2FSQfYayNHAJEDJTV0lXVVzkdccKtoVHiarCzcLLIyBRZT12dIrofoflGpbO4simR%2BohJhbaEfyw2VSjtiDFAVlieHo4PEUAnoGEfVtE2opS32tkOPJmHOvedcrrClw26b6Lwoq4tFpwQ0AAL5GtuopD0xN5YYMmSRJC9kBOkPdBSjxxOG7pO2TKSvQP1NXeposWqJ4WqFXFewdFbtrGwP5Lo31NfqfOb7gpzQtOXnYQUpMW%2BoF3LvOti0ehXZLSdLaxCA%2Bs6Wruwn0ISvuB7euuFJ0U754c5lqi067142oE4hhdBKfay019mDqdPTtG48iIIgc%2FCdyr%2FfYrBnNPIhFvMuC8%2FWrwsR1orfaeXqOdhvGel6PxWpWqMkMwC17DrbizjCiIRyg89m%2BDGfT2Qxb1Fwjry%2BtTZRoDPa6Et2sDxd7yj%2BxLaF0qu%2B%2FSXI4LpOyJpT129Ms4X3MIDEULHSc3dnlisNVqRuu9sxV1VZU3UNMlclQZYUOd6WbIAi%2FNeOGctSGvjJGOiLlJ1ZsNW9xocGQTuNwGZceRMxjN5BvRBvt2PzFbQ4oDghcRS23mOZFW%2FJnc6uF2BXOgQD5C4akAwDiGq97W3eGbkARPVCILIr7dimAIPIo5Xbg5UrCwaX9K%2FpuvXS8VZnd18CVttxgq1cmd%2FQWuCn2JRmcKpLU4vgOo%2FUQmvXpC15ksokzeM8phAsp5pWDsdS6CkxSSwk3MlAv3LR6DewlUiQdtxUx898HTX0XO766Jmq9kxdw%2BPUVcazYLDRB6XKhyQ0PkMrPM2FfqcrJ0nDhAplVoy6X8GTe73enErNuNilCiJTNdAc3TVjk60hF6pGvkPTuE7xi9oJEKkNrec%2FF6wkdTatG9ECl7StYG5V14qWyy7QASDLKyvHCmmgQnoZ6mxjhfSJCmnoG0P9Vkg3UwTGplMDbzrFMG4NBFgRX%2ByH2doLFDUT1yO%2BusXXRbmQRZe1XrOpWWfV1mNLxiGh6%2BKWjAz3UsEGIgC82uiZB%2BKrWUvGgRob59sWBUOlh4Cd0tFHKYoBO6XRZykUvmKdHdsZQke%2B1cSlygdRVfo7bTXRB9ALypnBaUItEDUFuyzznFR6t9IOEUrxLKX6LKF4llx9lnjWCjWt%2Bl6VhGywPgNoJwbQyqK6rvdd%2BRZy3y8Oex40Yll3718Q9CKwIQ1K7jMkCNl%2F91Nh7JF3ZKMDzSmcqLEo%2FRGhMlNXTLYwqlvPFBjT9xZK3U1YcmXSLyvSty%2FyC0RWnf3O4g0XFLPMWoU6dHXVKwHmqa19eukuMrPDDA1Cb6YTbKam6xZzNaJcWWlOy%2BYnYRYGTcOIGnxE%2F0eQvpUEjZ4qj4XCN1JUoJur1OFHGUF2XF5HdA%2Fs%2BNjwgXBY2iEYe%2FY%2FUbuZkOM%2Bmr69TNkyGXt6xs%2FRoU%2FItMKP8tw822WrBVQd4LmSAHYs6qxgGaoXuDlHypUSmlmzsWY%2ByTkOkpJb1faQVK616zrJ%2B25os%2FRQ1O7c3S%2BcDCdzPfsQHfjPd2oi%2FUS%2F7pGRQQ1p%2BmZkJ9xw94CwKgkaCsy7F6BigY9BFXe9Q68y1aQJYp6%2FUU7BloKnfuH34AZumoV1KgSZzRo8EQnvQEEwuJJC6dadysyXfZLob5mzVpZFToCXVu6BLp5S1dZYKJ5SkeMric0XphkVd6lSjgSeM%2BTT%2BzSAxGDQ5XyHinqesE%2Bmrpzb2eFiwlY5QTCO%2F9jUi9ohi%2BaGAM9JWtnd6toCwolr9JYsfEOd1iIJG39Lmn5e%2BmBbx%2BamVYKYaDtBXhHKu7diqZj1hcVDZky6NFszdK%2BxNL21LSukREjzylOfh4NIV5SejCM1Xpx0xChjApB1lNiZbfelhGFV2X5taLB6ADsbvCDCR73i%2BAg6Ajo5w3oZwQ11fO4VgMmFRwDeMQAZrpf2qbga6CqLskfQ3T7o%2BFKNU5OuDcBy19tAAfgFxVaGyI%2BoO0%2FWQinm%2FaKu0us2HNSd7%2FfN%2BQGR9dv5QO3bW6tInCAapVwr7fqXwY%2FWXbdmMJlauu5XVIeDz5o5J2MOSXRULeTzCVD%2BlN5Wkw%2Fyp4fpVh19MOTNbP7AFqIz%2Fg8%3D)


<iframe frameborder="0" style="width:100%;height:848px;" src="https://viewer.diagrams.net/?tags=%7B%7D&highlight=0000ff&edit=_blank&layers=1&nav=1&title=diagramm.xml#R7V3bcptIGn6WuVDV7oUozodLW46SbM3UeOKkUnuVQqIlsUFqBpDszNNvNydB8yNAAoRkchHbTQMN%2Ff3nAxNptn376Jnu5g9sIWci8tbbRHqaiKKgqgb5QUd%2BRSOywUcDa8%2B24knHgRf7HxQPJtP2toX83MQAYyew3fzgEu92aBnkxkzPw6%2F5aSvs5O%2FqmmtUGHhZmk5x9LttBZtoVFf44%2FgnZK83yZ0FPj6yNZPJ8YC%2FMS38mhmSPkykmYdxEP22fZshh7685L1E581LjqYL89AuqHOC%2BYfh%2FTpIbwdNdP%2FGb389ffn0YRpf5WA6%2B%2FiBJ6LqkOs9rjC5LHlh5jI6oP69pyt9nOHtCnuBaR6H6PaagTmlp0x9b5mbvwkCulUPdGHinE7xuTXGaweZru1zS7wlw0ufTJmvzK3tUJyA91DX9Oc3H3nJEuPrJYfi1xz8SvaOvHGX%2FrrfOr%2FbK%2BTYO%2FLXo4s8e4sCchnpyYmHn49jj68bO0AvbvTcrwTV4UNsHfKXQH4lQAtMcgo9nSd%2FWx52v5reGgXxwBI7jun69iJcBh3x0HLv%2BfYBfUF%2BBHA6ivcBvfUsBS4ddMljkxE%2F8Mg96CAy%2FeAV%2BfT5duj1g0XAmTyg9nh8PSg9cHxj9J2nx50Fzpydea0zZlLJBQ7IC2xCFiWnkgc8ICt3Lp%2Bb4OH9zirM0J4yh6L3W4R1glGyAvSWGYph%2FhFhsnXeLzIlPipLnCBFJyVcJ6bB1yMJp4S6yZCvnhCrGbONdXrxI2WRX2LiakBoIkBoDFwzIHMx2f1wCcrjRHliUEswssFrvDOdLG5BPAUhMl8oIfhFSniHYCvC6yRfrI85JQc4DQJcEW%2Ba0hXelCrG3gPHrik8Iu79SGX1yNvvjNxa5%2B0iL3K6UMncxV6Zu3ZrxEbePyU1%2FjtafLbqklyGUBLEPDj2ekfGFjgIyErpha0HqnXTMQcvf%2BbgSxCfh2oCS%2BkphWAKt5i8Uvzwp%2FDj470XP%2F1J8RskxFw2T43m0TWfRKOHHDMgVJ%2B3NwBcxac%2BU4F%2BRLFg5MWFIDHQjB4oPuuITvJuzV%2BZabGeUHqfVOgk91EYG4Fdl8rMl0%2FPT2kPvj75JVoxc3ayfLxa%2BSiYsOSXvufzKVJvpG7VgTN20Y6KBNPfpPyMHI4tVv0KQN8j5eXpBf81fVlMZ%2Fajo60%2B4uTBuwawqDIAVhkAR5R2MYAlFpBqLYAlSplYscoSMjsDhac243bkQiwP%2BGcPH2zrqI0tvGTGw57IWJF%2FQQS7wGGiBUUILZ8yanh3rOHFb0g5ycQaWFWyxuVJGFL2JEjZ41tQ9kCiNhqJltGS78%2BSP8mDa2OOlThGPVNeEFoA3M7531ravL18M5T%2FPqFv%2F3w7vO6nYjNlZkRct4grIKkuCE8gjpPyJq0hJmyvAnWJ26l1LidADstB6y4e%2BntP8ZdatfwTXu63KFQx7t7AjQ3XrIF7kpn0biCwns6WLFxJOcsCLTMQ2FWWmDEdmbIwJd6cd%2BlIiX9%2BnpFTaHB2Za%2FPNwXujDBPqZWd06WWpxexK7pkPENihSdJUhRwXTXpmL3bEOm40htAbE%2Fs5YhsIkp8%2BC9Hi0fKSAYj4g9tL0LrvMC7bxF5NSXg%2FgL5Hz98nVCH%2FNxCWzx1MRHYc9fDK5tqffOl6ZEBaYuaegsIiZqOgxy89swto%2FvmjjVwGqzsN5Q4%2BqK%2FHWeW7pW00pdouaTGb%2BDhnyhzZKErclumsGjAgi6rIYpQTLsNuwQEtHiDgmlt%2B1HkY%2BbYGc1wFEkDEklSR8EQiYnRJ%2Fc5T1UsrHIAIqahp%2BCOwh5GEdLXjO%2BxTqRUG2o7PMJoTZLcCNISfHZf4ZGEgkaNKJQxz3%2B%2BxCoRh%2B0l1YeIvBr1nwTqYomUyKV9pOGCXjQgSaoCcBPtxN7ZgenQ10yAH6snvBnG2zrRSXY49OS2yL99wtYCVuUJB%2Be2kyyNrCD%2Ba1gpHTHOeI7n4yVcmuTBM4g1utFrROY%2BMl%2BR5CE2nK8p4HPUEzKiUvEWBqA3SVBMrz9BNDQzKX9XHzu2pXA%2BYSSmMyX35Sx0CA%2BNkimiH0HnGBKBYoZ6v7JJrpRN7wrTzLMP9ZmixXIcV0pdt%2B8vsxSkWzJElbq4kFT1QqpM%2FGPCAGhQbosGz4G675q7M6EeXSjYutO9701DCTCR%2Ba1pO1wgbLktPfMr8gNBlH7LIC6643lpXvcHxPjoVJBY8SDLxei%2BmJnWCzgVyPpuxaLox1mUMzZiN1XW1uDr2xqMbVDDqBDhLb%2FQYmCywuWaqnJTg0HQNPA%2BdRX6Qk5eTcOmLXVdaZaCdztuzu7dkuzW1Q2UNve1S%2BB96kFMYSzaukHq1vzorTkmh6MGV%2BsBRokakEZquVfy7qY%2Fd%2FiVSPk5JRnbmka5HXuP4BLvbkDo9xSt1QtCH0pd7ixgC%2Bd%2BVaH6ltCql6A1ahBAKxioaURYVL7IrXmkN5%2FgXgG%2BEpwAaCp3dKscE2rUgVzQEm2xjRJHEDuV3oQ7ws4HWh1JkfM7XtvUhJp5yPJvCEHF%2FgcqYAx3CKFTjvgxjX0QaeyXibeCDSCI9RLWRa2rjHV5LMy5Y3zJIsDBIOWJLfFsD18j%2F7obfCl8AV9gi6Be8VWZTtnECUwAsfO3Nj3NibUYqqF1lEoQxvq%2Fbuxdu46SnIsvdi6Wu%2FhOdhjp37Ui6zSVIPNPzV%2BzraxGNvofOxFLE%2B2ZlhCyXjHfYOc38%2BQUnYWMAjCA%2BL9y3Vjp0BLRyGTbcsn%2FMecYQ%2F3HVJsCnFVI7%2B01jlMZZOwyeFhmStM%2BEeQ9Ex4f0Hy2bNtG0IA%2BJ3R4hl19YV%2FB%2FN6nburs3iu97v1YUwQyrtAp7W1H1nWKdSlQhkRXrMvfLIld8pNoerP51zfnO7%2F7tAe7z16miA4yjpdPZgXfhNSRUlqIQiiSwWmMJto45leqLCp8rRt2HR%2BubH55DYn4Ja2KnkXcKY6Y8XNMnc5JQRr%2FsFwi%2F7S3%2BSaEpViEAuTD0cCUrDac0CChiW1CIxUz79SorizLSnyylaWGcm%2BFWUVU1mwK19gwl%2FKGs6JVVMDrOiugGzW7g3hvza58HdnaIAEOkjd%2F3tmBHdbeWGTWV6KF0ct9QZbtUUfszfNiRWBjyooG2K2i2FV6LAiFcoenZR9qb%2FHCXP5ch290Ghs7dKftaEcjAyQz7q0X%2F4o8zXzy49%2BxMUOhklgZD6GYTnB3PF4ja3aiPEq8RF3%2BSS2ftSQ%2Fo0IuYo%2FIE%2B0xKu%2F64cUA%2B7H3bOrbPo0qEH65N%2FXebBpB4TlVqsR1NqWin84ILX1Wo2UmZ1GHzIo8nmpu6XbuFj79kWqeX8L2CEkmF%2F9hZ7mRWLgpZifoeU4H9uKHvDRt6Jxw55cbasbfBryStk%2BffX9%2FSQfYayNHAJEDJTV0lXVVzkdccKtoVHiarCzcLLIyBRZT12dIrofoflGpbO4simR%2BohJhbaEfyw2VSjtiDFAVlieHo4PEUAnoGEfVtE2opS32tkOPJmHOvedcrrClw26b6Lwoq4tFpwQ0AAL5GtuopD0xN5YYMmSRJC9kBOkPdBSjxxOG7pO2TKSvQP1NXeposWqJ4WqFXFewdFbtrGwP5Lo31NfqfOb7gpzQtOXnYQUpMW%2BoF3LvOti0ehXZLSdLaxCA%2Bs6Wruwn0ISvuB7euuFJ0U754c5lqi067142oE4hhdBKfay019mDqdPTtG48iIIgc%2FCdyr%2FfYrBnNPIhFvMuC8%2FWrwsR1orfaeXqOdhvGel6PxWpWqMkMwC17DrbizjCiIRyg89m%2BDGfT2Qxb1Fwjry%2BtTZRoDPa6Et2sDxd7yj%2BxLaF0qu%2B%2FSXI4LpOyJpT129Ms4X3MIDEULHSc3dnlisNVqRuu9sxV1VZU3UNMlclQZYUOd6WbIAi%2FNeOGctSGvjJGOiLlJ1ZsNW9xocGQTuNwGZceRMxjN5BvRBvt2PzFbQ4oDghcRS23mOZFW%2FJnc6uF2BXOgQD5C4akAwDiGq97W3eGbkARPVCILIr7dimAIPIo5Xbg5UrCwaX9K%2FpuvXS8VZnd18CVttxgq1cmd%2FQWuCn2JRmcKpLU4vgOo%2FUQmvXpC15ksokzeM8phAsp5pWDsdS6CkxSSwk3MlAv3LR6DewlUiQdtxUx898HTX0XO766Jmq9kxdw%2BPUVcazYLDRB6XKhyQ0PkMrPM2FfqcrJ0nDhAplVoy6X8GTe73enErNuNilCiJTNdAc3TVjk60hF6pGvkPTuE7xi9oJEKkNrec%2FF6wkdTatG9ECl7StYG5V14qWyy7QASDLKyvHCmmgQnoZ6mxjhfSJCmnoG0P9Vkg3UwTGplMDbzrFMG4NBFgRX%2ByH2doLFDUT1yO%2BusXXRbmQRZe1XrOpWWfV1mNLxiGh6%2BKWjAz3UsEGIgC82uiZB%2BKrWUvGgRob59sWBUOlh4Cd0tFHKYoBO6XRZykUvmKdHdsZQke%2B1cSlygdRVfo7bTXRB9ALypnBaUItEDUFuyzznFR6t9IOEUrxLKX6LKF4llx9lnjWCjWt%2Bl6VhGywPgNoJwbQyqK6rvdd%2BRZy3y8Oex40Yll3718Q9CKwIQ1K7jMkCNl%2F91Nh7JF3ZKMDzSmcqLEo%2FRGhMlNXTLYwqlvPFBjT9xZK3U1YcmXSLyvSty%2FyC0RWnf3O4g0XFLPMWoU6dHXVKwHmqa19eukuMrPDDA1Cb6YTbKam6xZzNaJcWWlOy%2BYnYRYGTcOIGnxE%2F0eQvpUEjZ4qj4XCN1JUoJur1OFHGUF2XF5HdA%2Fs%2BNjwgXBY2iEYe%2FY%2FUbuZkOM%2Bmr69TNkyGXt6xs%2FRoU%2FItMKP8tw822WrBVQd4LmSAHYs6qxgGaoXuDlHypUSmlmzsWY%2ByTkOkpJb1faQVK616zrJ%2B25os%2FRQ1O7c3S%2BcDCdzPfsQHfjPd2oi%2FUS%2F7pGRQQ1p%2BmZkJ9xw94CwKgkaCsy7F6BigY9BFXe9Q68y1aQJYp6%2FUU7BloKnfuH34AZumoV1KgSZzRo8EQnvQEEwuJJC6dadysyXfZLob5mzVpZFToCXVu6BLp5S1dZYKJ5SkeMric0XphkVd6lSjgSeM%2BTT%2BzSAxGDQ5XyHinqesE%2Bmrpzb2eFiwlY5QTCO%2F9jUi9ohi%2BaGAM9JWtnd6toCwolr9JYsfEOd1iIJG39Lmn5e%2BmBbx%2BamVYKYaDtBXhHKu7diqZj1hcVDZky6NFszdK%2BxNL21LSukREjzylOfh4NIV5SejCM1Xpx0xChjApB1lNiZbfelhGFV2X5taLB6ADsbvCDCR73i%2BAg6Ajo5w3oZwQ11fO4VgMmFRwDeMQAZrpf2qbga6CqLskfQ3T7o%2BFKNU5OuDcBy19tAAfgFxVaGyI%2BoO0%2FWQinm%2FaKu0us2HNSd7%2FfN%2BQGR9dv5QO3bW6tInCAapVwr7fqXwY%2FWXbdmMJlauu5XVIeDz5o5J2MOSXRULeTzCVD%2BlN5Wkw%2Fyp4fpVh19MOTNbP7AFqIz%2Fg8%3D"></iframe>

## Jupyter Notebook

- For a hands-on demonstration of the reverse-engineered authentication using the Python `requests` library, access the Jupyter Notebook here: [Presentation: Reverse-Engineered Authentication Notebook](https://colab.research.google.com/drive/17H9eaPhxgqb4yonFpfEle39pQjWghL_o?usp=sharing)


<link rel="stylesheet" href="https://github.githubassets.com/assets/gist-embed-f65f23c5975e.css">
<div id="gist126680771" class="gist">
    <div class="gist-file" translate="no">
      <div class="gist-data">
        <div class="js-gist-file-update-container js-task-list-container file-box">
          <div id="file-copy-of-reverseengineeredsolidauthenticationbytimschupp-ipynb" class="file my-2">
            <div itemprop="text" class="Box-body p-0 blob-wrapper data type-jupyter-notebook">
              <div class="render-wrapper">
                <div class="render-container is-render-pending js-render-target"
                     data-identity="b896a2c7-130f-40d6-9e79-f19c42218f0a"
                     data-host="https://notebooks.githubusercontent.com"
                     data-type="ipynb">
                  <svg style="box-sizing: content-box; color: var(--color-icon-primary);" width="64" height="64" viewBox="0 0 16 16" fill="none" data-view-component="true" class="octospinner mx-auto anim-rotate">
                    <circle cx="8" cy="8" r="7" stroke="currentColor" stroke-opacity="0.25" stroke-width="2" vector-effect="non-scaling-stroke" fill="none"></circle>
                    <path d="M15 8a7.002 7.002 0 00-7-7" stroke="currentColor" stroke-width="2" stroke-linecap="round" vector-effect="non-scaling-stroke"></path>
                  </svg>
                  <div class="render-viewer-error">Sorry, something went wrong. <a class="Link--inTextBlock" href="https://gist.github.com/user-1024/00f76dfe47a63659de070d449c9946f9.js">Reload?</a></div>
                  <div class="render-viewer-fatal">Sorry, we cannot display this file.</div>
                  <div class="render-viewer-invalid">Sorry, this file is invalid so it cannot be displayed.</div>
                  <iframe class="render-viewer"
                          src="https://notebooks.githubusercontent.com/view/ipynb?bypass_fastly=true&amp;color_mode=auto&amp;commit=db9efc58d843777a23609c871a7478f9a282b6c4&amp;docs_host=https%3A%2F%2Fdocs.github.com&amp;enc_url=68747470733a2f2f7261772e67697468756275736572636f6e74656e742e636f6d2f676973742f757365722d313032342f30306637366466653437613633363539646530373064343439633939343666392f7261772f646239656663353864383433373737613233363039633837316137343738663961323832623663342f636f70792d6f662d72657665727365656e67696e6565726564736f6c696461757468656e7469636174696f6e627974696d7363687570702e6970796e62&amp;logged_in=false&amp;nwo=user-1024%2F00f76dfe47a63659de070d449c9946f9&amp;path=copy-of-reverseengineeredsolidauthenticationbytimschupp.ipynb&amp;repository_id=126680771&amp;repository_type=Gist#b896a2c7-130f-40d6-9e79-f19c42218f0a"
                          sandbox="allow-scripts allow-same-origin allow-top-navigation"
                          title="File display"
                          name="b896a2c7-130f-40d6-9e79-f19c42218f0a">
                    Viewer requires iframe.
                  </iframe>
                </div>
              </div>
            </div>
          </div>
        </div>
        <div class="gist-meta">
          <a href="https://gist.github.com/user-1024/00f76dfe47a63659de070d449c9946f9/raw/db9efc58d843777a23609c871a7478f9a282b6c4/copy-of-reverseengineeredsolidauthenticationbytimschupp.ipynb" style="float:right" class="Link--inTextBlock">view raw</a>
          <a href="https://gist.github.com/user-1024/00f76dfe47a63659de070d449c9946f9#file-copy-of-reverseengineeredsolidauthenticationbytimschupp-ipynb" class="Link--inTextBlock">
            copy-of-reverseengineeredsolidauthenticationbytimschupp.ipynb
          </a>
          hosted with ❤ by <a class="Link--inTextBlock" href="https://github.com">GitHub</a>
        </div>
      </div>
    </div>
</div>

Exploring the workings of OpenID Connect, the Solid Community Server implementation, and reverse-engineering client authentication with Python requests.

Understanding Solid Authentication


## Intro 

Sentry is an incredibly nice error tracking tool; it can significantly increase the speed of bug fixing and debugging. It also offers a bunch of other features. You should also check out [their service](https://sentry.io)

Since I like to know how the stuff works and willing to take it appart; I would always prefer to self-host if I can. This also gives me more comfort about controlling my own data which by the way also aids GDPR compliance (though this is also possible using Sentry's service directly).

Fortunately, there [still is a Helm chart](https://github.com/sentry-kubernetes/charts/tree/develop). 
'Still', because for example: [the deprecated Posthog chart](https://github.com/PostHog/charts-clickhouse), but also the originally (officially maintained?) [sentry chart](https://github.com/helm/charts/tree/master/stable/sentry) seems deprecated now.

## Setup

I prepared a script that will:

1. Configure an ingress with a cluster issuer annotation to serve under a specific `$HOSTNAME`
2. Configure a Postgres password
3. Enable session recordings and its interface

## Installation

Use the following script to install Sentry:

```bash
./install_sentry_microk8s.sh \
  K8_NAMESPACE="<installation-namespace>" \
  RELEASE_NAME="<release-name>" \
  SENTRY_HOSTNAME="<...>" \
  SENTRY_HOSTNAME="<...>" \
  POSTGRES_PASSWORD="<...>" \
  SMTP_USERNAME="<...>" \
  SMTP_PASSWORD="<...>" \
  SMTP_PORT="<...>" \
  SMTP_FROM_EMAIL="<...>" \
  SENTRY_ADMIN_EMAIL="<...>" \
  SENTRY_ADMIN_PASSWORD="<...>"
```

I found out about having to set and update the admin Postgres password using a secret; [in this Github issue](https://github.com/sentry-kubernetes/charts/issues/554).
That's why we create a simple secret for the Postgres-password here:

```bash
apiVersion: v1
kind: Secret
metadata:
  name: postgres
  namespace: $K8_NAMESPACE
stringData:
  postgres-password: "$POSTGRES_PASSWORD"
```

For enabling the session feature and directly setting the Python config, I read [over here](https://github.com/getsentry/self-hosted/issues/2057) and [here](https://github.com/getsentry/self-hosted/blob/master/sentry/sentry.conf.example.py) and [here](https://github.com/sentry-kubernetes/charts/issues/918), ending up with:

```yaml
config:
  sentryConfPy: |
    SENTRY_FEATURES["organizations:session-replay"] = True
    SENTRY_FEATURES["organizations:session-replay-ui"] = True
```

Ingress setup follows a pretty standard procedure:

```yaml
nginx:
  ingress:
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    enabled: true
    hostname: $SENTRY_HOSTNAME
    ingressClassName: "public"
    tls: true
    extraTls:
      - hosts:
          - $SENTRY_HOSTNAME
        secretName: sentry-tls
```

As in some of my other scripts, I like to assume the cluster issuer is named `"letsencrypt-prod"`. To me, that's a reasonable sacrifice for adding fewer inputs ;).

Also, **to get the install to succeed** I needed to increase the `hooks.activeDeadlineSeconds`; otherwise, it would time out and the installation would fail.

```yaml
hooks:
  activeDeadlineSeconds: 3500
```

## Open Issues

Currently, I still experience one of the metrics services failing. I'm not sure about the impact yet, but tracking seems to work just fine:

```bash
microk8s kubectl logs sentry-snuba-metrics-consumer -n sentry
snuba.clickhouse.errors.ClickhouseWriterError: Method write is not supported by storage Distributed with more than one shard and no sharding key provided (version 21.8.13.6 (official build))
```

seems like an issue with my storage provisioning possibly; no impact seen so-far.


Self Hosted Error Tracking and Reporting using Sentry Kubernetes and Helm

Self Host Sentry using Helm


Since launching [Msgmate.io](https://msgmate.io) at the end of March, we've seen a lot of activity and received a myriad of feedback regarding its services and development.

However, the development of the direct Msgmate feature has been quite restricted. As a result, the updates that are visible to the public have also been very limited. Therefore, I am writing this article to shed some light on what's in store for Msgmate's future.

Please note that this article is somewhat technical in nature!

### Statistics

<div style="
    display: flex;
    flex-wrap: wrap;
    justify-content: space-around;
">
    <div style="
        background-color: #2C333A;
        padding: 20px;
        border-radius: 10px;
        width: 20%;
        text-align: center;
        min-width: 320px;
    ">
        <h2 style="color: #E88388; font-size: 24px;">
            <strong>Registered Users:</strong>
        </h2>
        <p>300</p>
    </div>
    <div style="
        background-color: #2C333A;
        padding: 20px;
        border-radius: 10px;
        width: 20%;
        text-align: center;
        min-width: 320px;
    ">
        <h2 style="color: #E88388; font-size: 24px;">
            <strong>Messages Exchanged:</strong>
        </h2>
        <p>6,331</p>
    </div>
    <div style="
        background-color: #2C333A;
        padding: 20px;
        border-radius: 10px;
        width: 20%;
        text-align: center;
        min-width: 320px;
    ">
        <h2 style="color: #E88388; font-size: 24px;">
            <strong>Tokens Used:</strong>
        </h2>
        <p>579,807+</p>
    </div>
    <div style="
        background-color: #2C333A;
        padding: 20px;
        border-radius: 10px;
        width: 20%;
        text-align: center;
        min-width: 320px;
        margin-bottom: 0 !important;
    ">
        <h2 style="color: #E88388; font-size: 24px;">
            <strong>Images Generated:</strong>
        </h2>
        <p>192</p>
    </div>
</div>

Since the alpha launch in March, we've had 300 users register. A total of 6,331 messages have been exchanged, using over 579,807+ tokens for prompts and completions. Furthermore, a total of 192 images have been generated.

> I've received feedback indicating interest in such a service.
> However, there were also critical comments concerning the current reliability and limited range of features available.

Rest assured, I am taking all feedback seriously and am actively working to bring you a better and enhanced user experience.

<strong style="color: red;">For now we limited msgmate.io alpha registrations untill we scaled our bot service to handle more chats in paralel</strong>


### Development in the Software Stack

My interest in large language models and their immediate applications in everyday life led me to initiate Msgmate as an experiment. Having considerable experience in development operations and infrastructure, I've managed to evolve my stack through several iterations based on the backend + infrastructure definitions I've developed over the years.

**Here's a look at the evolution of the stack:**

|Iteration|Link|Description|
|--|--|--|
|1|[Django Clean Slate](https://github.com/tbscode/django-clean-slate)|First stack iteration which involved Django + Webpack + React.|
|2|[Django NextJS Setup](https://github.com/tbscode/django_nextjssetup)|First experimental combination of Django + NextJS + React.|
|3|[Tiny Django](https://github.com/tbscode/tiny-django)|Evolution into the first version of 'Tim's Stack' initially dubbed 'tiny-django'.|
|4|[AnySearch Stack](https://github.com/tbscode/anysearch-stack)|First Live application using the stack for a submission to the tum.ai hackathon. The stack proved to be instantly viable for rapid development and easy deployment to microk8s Kubernetes clusters.|
|5|[Tim's Stack Anystack](https://github.com/tbscode/tims-stack-anystack)|Clean re-write of the stack, adding automation documentation, and more. During the Bunnyshell Jamstack Hackathon, I had the unique opportunity to|
|6|[Tim's Stack V2](https://github.com/tbscode/tims-stack-v2)|Dawn of a new age ;) Finalization work on the stack has started with the objective of making it out-of-the-box ready, usable state.|

All these updated work towards shaping the backend that msgmate.io uses and making it more robust and scalable! The bigger beta update planned for msgmate will be using the update Tims Stack V2 and there are plans for launching an android and ios app along side the new backend and interface.

### The Stack Won 2. Place!

As mentioned in the Above list [I've participated in a Hackathon using my Stack](https://blog.t1m.me/blog/tims-stack). 

> 🥳 Checkout [the hackathon page](https://devpost.com/software/tim-s-stack-anystack).

My efforts in the Bunnyshell hackathon didn't go to waste, as we managed to secure second place and win `$4000`, which I'll be reinvesting in the development of Msgmate.io!

### Msgmate.io Vision

I envision msgmate.io as the premier platform for innovators and casual users a like - who wish to engage with large language models - automated agents - and seamlessly incorporate them into their applications and everyday life. 
Msgmate should provide user-friendly developer APIs, websockets, and client applications. Fundamentally, interaction with language models and agents should be as straightforward as possible. Yet beneath the surface, the platform should offer abundant interfaces and configuration options.

### Development in the Team

Given my commitments to other projects, the time I can invest in Msgmate development is limited.
So, to advance this service further, I need more development power!

Using the prize money from the hackathon, I've already hired a NextJS + React developer to help me revamp the Msgmate interface. 

**However, I still need more developers motivated to work on LLm tooling and software.**

> I am looking for motivated (self) taught developers or students who have experience working on their own projects.
> You need to be a fast learner. Having knowledge of Docker, NextJS, React, Django would be advantageous but not a requirement.
> If you are interested in collaborating with us, get in touch at tim.timschupp@gmail.com

### Want to Stay Updated? Have Feedback or Issues to Report?

We've set up a simple form over at [`msgmate.io/newsletter`](https://msgmate.io/newsletter/). By signing up, you'll receive the next update straight to your inbox.

Feel free to add a comment to the signup request for any feedback or bugs. You can fill out the newsletter request as often as you like! Happy messaging with Msgmate.io!


A progress and state update on the development of my msgmate.io plattform

Msgmate.io Development Update Oct 2023


Different command I've used over the year to quickly create pdfs from local note files.

## Raw Pandoc

```bash
pandoc -V geometry:margin=1cm <mardown-file> -o <pdf-file>
```

## Pandoc + `wkhtmltopdf`-engine

- html support, e.g.: tables etc
- dynmaic margin

```bash
pandoc --pdf-engine=wkhtmltopdf -V papersize=a4 -V margin-top=0.1 -V margin-left=0.1 -V margin-right=0.1 -V margin-bottom=0.1 --from markdown-markdown_in_html_blocks+raw_html <markdown-file> -o <pdf-file>
```

## Quick TXT file to dense readable pdf

```bash
pandoc --wrap=preserve $1 -V geometry:margin=1cm -V fontsize=8pt -o $1.pdf -f markdown+hard_line_breaks
```

## Github markdown using `grip`

```bash
sudo snap install grip
grip <markdown-file>
```

then visit the browser page and export the page as pdf.
Or `curl -X <page> > rendered.html`.

## HTML to pdf with

```bash
pandoc --pdf-engine=wkhtmltopdf rendered.html -o <pdf-file>
```

## TXT to pdf with `enscript` and `ps2pdf`

```bash
find . -type f -name $1 | while read ONELINE; do enscript "$ONELINE" -p "$(echo "$ONELINE" | sed 's/.txt/.ps/g')"; done
find . -type f -name "$(echo "$1" | sed 's/.txt/.ps/g')" | while read ONELINE; do ps2pdf "$ONELINE" "$(echo "$ONELINE" | sed 's/.ps/.pdf/g')"; done
```

A collection of strategies to quickly produce pdf's from markdown files

PDF's from markdown, pandoc and more


Simulate a webcam even in your deveice doesn't have one!
The video input can even work to simulate video inputs on remote development devices.

You'll need to install and anable `v4l2loopback`.

```bash
sudo apt install v4l2loopback-dkms
sudo modprobe v4l2loopback
```

## Live video from screen

```bash
sudo ffmpeg -f x11grab -r 15 -s 1280x720 -i :1 -vcodec rawvideo -pix_fmt yuv420p -threads 0 -f v4l2 /dev/video2
```

## From Video File


```bash
ffmpeg -re -i <some-video>.mp4 -map 0:v -f v4l2 /dev/video2
```

> Where to go from here? Loop a video of you listen patiently in your zoom meetings? ;)

How to use `ffmpeg` to funnel screen or video data as video input.

Simulate secondary webcam on linux


This guide illustrates the steps required to build, push, and deploy a Next.js app on Kubernetes using docker files for defining development and production Next.js images. These docker compose files allow for the building and pushing of images, with a Next.js config able to export static files. A simple helm chart will create deployment, service, and ingress for exposing our app.

> Template repo containing all the required code and configs can be [found on my github](https://github.com/tbscode/nextjs-helm-template).

### Prerequisites

This tutorial can also be adapted to work with any managed cluster as I've been using similar setups to quickly deploy Next.js apps for a while.

For your cluster you'll need:

- `cert-manger` cluster issuer with name `letsencrypt-prod` setup.
- `nginx-ingress` installed.
- `helm` installed.
- Public IP and Host Url connected.

Further, a private container registry is required.

> If you don't have a Kubernetes cluster ready follow [My Blog Post on Microk8s Private Cluster Setup](/blog/microk8s-on-vps).

### 1. Start a Next.js App

Begin by creating a simple Next.js app:

```
npx create-next-app@latest
What is your project named?  frontend
Would you like to use TypeScript? Yes
Would you like to use ESLint?  Yes
Would you like to use Tailwind CSS?  Yes
Would you like to use `src/` directory?  No
Would you like to use App Router? (recommended)  Yes
Would you like to customize the default import alias (@/*)?  No
```

We named the app `./frontend` so the respective folder is created.

### 2. Dockerize the Application

Two docker files can be created:

1. [`./frontend/Dockerfile`](https://github.com/tbscode/nextjs-helm-template/blob/main/frontend/Dockerfile): A development docker file that mounts frontend `./frontend` into the container for hot-reload.
2. [`./frontend/pro.dockerfile`](https://github.com/tbscode/nextjs-helm-template/blob/main/frontend/prod.dockerfile): A production docker file that statically builds the entire application and Next.js in a container.

The development docker file is straightforward:

```Dockerfile
FROM node:16-alpine

WORKDIR /frontend
COPY ./package.json .

RUN apk add curl

RUN npm i --save-dev

ENTRYPOINT ["npm", "run", "dev"]
```

Here, we execute `npm i` and then `npm run dev`. When mounted to the host, the `./node_modules` directory will also be present on the host.

The production docker file, which is based on the Next.js example docker file, can be found [here.](https://nextjs.org/docs/pages/building-your-application/deploying#docker-image)

```dockerfile
FROM node:16-alpine AS deps
RUN apk add --no-cache libc6-compat curl
WORKDIR /app

COPY package.json yarn.lock* package-lock.json* pnpm-lock.yaml* ./
RUN \
    if [ -f yarn.lock ]; then yarn --frozen-lockfile; \
    elif [ -f package-lock.json ]; then npm i; \
    elif [ -f pnpm-lock.yaml ]; then yarn global add pnpm && pnpm i --frozen-lockfile; \
    else echo "Lockfile not found." && exit 1; \
    fi

FROM node:16-alpine AS builder
WORKDIR /app
COPY . .
RUN apk add --no-cache curl
RUN npm i
RUN npm install --unsafe-perm -g sharp

ENV NEXT_TELEMETRY_DISABLED 1

RUN yarn build

FROM node:16-alpine AS runner
WORKDIR /app

ENV NODE_ENV production
ENV NEXT_TELEMETRY_DISABLED 1

RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs

COPY --from=builder /app/styles ./styles
COPY --from=builder /app/public ./public

COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static

USER nextjs

EXPOSE 3000

ENV PORT 3000

CMD ["node", "server.js"]
```

### 3. Compose Files for Development

Creating two compose files can pave the way for convenience:

1. [`docker-compose.yaml`](https://github.com/tbscode/nextjs-helm-template/blob/main/docker-compose.yaml): A development compose file.
2. [`docker-compose.pro.yaml`](https://github.com/tbscode/nextjs-helm-template/blob/main/docker-compose.pro.yaml): A production/ Local Infrastructure compose file.

Development is now always as straightforward as executing `docker-compose up`.

### 4. Creating a Helm Chart

To effectively manage the deployments and services we require, we prefer to create a simple Helm chart that can easily be adapted, installed, and updated. Our Helm chart will need the following components:

1. Frontend Deployment
2. Registry Image Pull Secrets
3. Frontend Service
4. Ingress

```
microk8s helm create ./helm
cd ./helm/template && rm -rf *
```

Everything will be deployed to a `rootNamespace`.

We have reset the template and created a simple setup for now [`./helm/template/frontend.yaml`](https://github.com/tbscode/nextjs-helm-template/blob/main/helm/templates/frontend.yaml).

```yaml
{{- if .Values.registryAuth.use }}
kind: Secret
type: kubernetes.io/dockerconfigjson
apiVersion: v1
metadata:
  name: dockerconfigjson-github-com
  namespace: {{ .Values.rootNamespace }}
stringData:
  .dockerconfigjson: >
    {{
      (
        dict "auths"
        (
          dict {{ .Values.registryAuth.registry }}
          (
            dict "auth" .Values.registryAuth.token
          )
        )
      )
      |
      toJson
    }}
{{- end }}
```

We've created a simple flag `.Values.registryAuth.use` to check if registry authentication should be used (as we wouldn't need it for local microk8s deployment).
We use `.Values.registryAuth.use` to define the registry that should be used.

Now we define a simple deployment for our nextjs backend:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-container
  namespace: {{ .Values.rootNamespace }}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend-container
  template:
    metadata:
      labels:
        app: frontend-container
    spec:
      containers:
        - name: frontend-container
          image: {{ .Values.frontend.imageURL }}
          ports:
            - containerPort: 8000
          envFrom:
            - secretRef:
                name: frontend-secrets
      {{- if .Values.registryAuth.use }}
      imagePullSecrets:
        - name: dockerconfigjson-github-com
      {{- end }}
```

Note that we inject the `imageURL` and the registry pull secret if required. We go on to expose the port `3000` to the cluster through a simple service.

```yaml
apiVersion: v1
kind: Service
metadata:
  name: frontend-service
  namespace: {{ .Values.rootNamespace }}
  labels:
    app: frontend-container
spec:
  type: ClusterIP
  ports:
    - port: 3000
      targetPort: 3000
  selector:
    app: frontend-container

```

It's also advisable to create a secret that injects some general environment variables listed in `values.yaml:frontend.env.*`

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: frontend-secrets
  namespace: {{ .Values.rootNamespace }}
type: Opaque
data:
{{- range $key, $value := .Values.frontend.env }}
  {{ $key }}: {{ $value | b64enc }}
{{- end }}
```

Finally, we set up an ingress:

```yaml
{{- if .Values.ingress.use }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend-ingress
  namespace: {{ .Values.rootNamespace }}
  annotations:
    nginx.ingress.kubernetes.io/proxy-connect-timeout: '3600'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '3600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '3600'
    nginx.ingress.kubernetes.io/server-snippets: |
      location /ws/ {
        proxy_http_version 1.1;
        proxy_redirect off;
        proxy_buffering off;
      } 
    {{- if .Values.ingress.certManager }}
    cert-manager.io/cluster-issuer: letsencrypt-prod
    {{- end }}
    kubernetes.io/ingress.class: public
spec:
  {{- if .Values.ingress.certManager }}
  tls:
    - hosts:
      - {{ .Values.ingress.host }}
    secretName: frontend-ingress-tls
  {{- end }}
  rules:
  - host: {{ .Values.ingress.host }}
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend-service
            port:
              number: 3000
{{- end }}
```

We've introduced some flags to set up `tls` if required and some annotations to facilitate websocket forwards.

#### Configuration via `Values.yaml`

Finally, to tie this all together, we create some default [`values.yaml`](https://github.com/tbscode/nextjs-helm-template/blob/main/helm/values.yaml)

```yaml
rootNamespace: "default"
registryAuth:
  registry: "localhost:32000"
  use: false
  token: ""
ingress:
  use: true
  certManager: false
  host: "localhost"
frontend:
  imageURL: "localhost:32000/frontend:registry"
  env:
    HOSTNAME: "localhost"
```

This can be used to control all the configuration options we just created.

### 5. Testing on Local Microk8s

It's always good to ensure that we can also deploy the entire infrastructure locally. For this, we can use a simple local `microk8s` installation.

```
microk8s enable ingress dns registry
docker-compose -f docker-compose.pro.yaml build
docker-compose -f docker-compose.pro.yaml push
```

Since we set up the template `.env` earlier, we're already configured to push and pull the image from `localhost:32000/frontend:registry`.
So now, all we need to do is install our newly created chart:

```
microk8s helm install next-js-frontend ./helm/ --set rootNamespace="nextjsfront"
```

Since we also set up `$ROOT_URL="http://localhost"`, the ingress will be configured to route to `http://localhost`. Now let's check the browser:

![Next page on localhost via microk8s](/static/assets/nextjs-page.jpg)

### 6. Deploy to Private Cluster

For the capability to deploy to a private cluster, you'll need a package registry available.

> If you quickly want a cheap registry, you can [check out my blog post](/blog/microk8s-gittea) on deploying Gitea in a private k8s cluster.

#### Modify Environment for Production

Firstly, we need to modify the environment to push to our private registry with a modified tag.

```bash
rm .env
echo "FRONTEND_IMAGE=\"<your-frontend-image-url>\"" >> .env
echo "ROOT_URL=\"<your-host-url>\"" >> .env
```

An example of an image URL is `my-gitea.example.com/gitea-private-user/package:latest`.
A `ROOT_URL` could be something like `nextjs.example.com`.

Note that you can restore the old default `.env` at any time using `git checkout .env`.

We also need to encode access credentials to our registry so we can inject them into our secret.

```bash
echo "<your-registry-user>:<your-registry-password>" | base64
```

```bash
docker-compose -f docker-compose.pro.yaml build
docker-compose -f docker-compose.pro.yaml push
KUBECONFIG="./kubeconfig.yaml" microk8s helm install next-js-frontend ./helm/ \
  --set rootNamespace="nextjsfront" \
  --set ingress.use=true \
  --set frontend.imageURL="<your-frontend-image-url>" \
  --set certManager.use=true \
  --set registryAuth.use=true \
  --set registryAuth.registry="<your-registry-host>" \
  --set registryAuth.token="<base64-encoded-token>"
```

> It works perfectly! In fact, this blog was deployed using this exact technique. 

### 7. Repository Automations

Now, we have a simple process to deploy the repository using GitHub Actions. 
These actions can also run on your own runners using Gitea Actions.

We create one simple workflow [`.github/workflow/deploy.yaml`](https://github.com/tbscode/nextjs-helm-template/blob/main/.github/workflow/deploy.yaml)

```yaml
name: Deploy NextJs App
    
on:
  workflow_dispatch:

permissions:
  contents: read
  pages: write
  id-token: write

jobs:
  deploy:
    runs-on: ubuntu-latest
    name: Deploy NextJS App
    steps:
      - uses: actions/checkout@master
        with:
          token: ${{ secrets.BOT_PAT }}
          submodules: recursive
          ref: ${{ github.event.pull_request.head.sha }}
      - name: Setup Kubeconfig
        id: setup_kubeconfig
        run: |
          touch .env; rm .env
          echo "${{ secrets.KUBE_CONFIG_BASE64 }}" | base64 -d > kubeconfig.yaml
      - name: Setup Environment
        id: setup_env
        run: |
          touch .env; rm .env; touch .env
          RANDOM_STRING="$( openssl rand -hex 3 )"
          FRONTEND_IMAGE="${{ secrets.CONTAINER_REGISTRY }}/${{ secrets.GITTEA_USER }}/nextjs-app:build-$RANDOM_STRING"
          echo "FRONTEND_IMAGE=\"$FRONTEND_IMAGE\"" >> .env
          echo "frontend_image=$FRONTEND_IMAGE" >> $GITHUB_OUTPUT
      - name: Build And Push Image
        run: |
          echo ${{ secrets.GITTEA_PASSWORD }} | docker login ${{ secrets.CONTAINER_REGISTRY }} -u ${{ secrets.GITTEA_USER }} --password-stdin
          DOCKER_BUILDKIT=1 docker-compose -f docker-compose.pro.yaml build
          DOCKER_BUILDKIT=1 docker-compose -f docker-compose.pro.yaml push
      - uses: azure/setup-helm@v3
        with:
           version: 'latest'
           token: ${{ secrets.BOT_PAT }}
        id: install-helm
      - uses: azure/setup-kubectl@v3
        with:
           version: 'latest'
        id: install-kubectl
      - name: Setup Cluster Connection
        id: cluster_connection
        run: |
          echo "Test"
          REGISTRY_AUTH=$(echo -n "${{ secrets.GITTEA_USER }}:${{ secrets.GITTEA_PASSWORD }}" | base64)
          KUBECONFIG=./kubeconfig.yaml kubectl create namespace nextjsnamespace || true
          cat << EOF | while read eval_command; do yq -i eval "$eval_command" ./helm/values.yaml; done
            .rootNamespace = "nextjsnamespace"
            .ingress.host = "${{ secrets.BOT_PAT }}"
            .frontend.imageURL = "${{ steps.setup_env.outputs.frontend_image }}"
            .ingress.certManager = true
            .registryAuth.token = "$REGISTRY_AUTH"
            .registryAuth.use = true
          EOF
          KUBECONFIG=./kubeconfig.yaml helm upgrade next-js-app ./helm/ --set rootNamespace="nextjsnamespace" --install
```

Using the following secrets configured in your GitHub account:

- `BOT_PAT`: GitHub bot access token
- `GITTEA_USER`: Private registry user
- `GITTEA_PASSWORD`: Private registry password or auth token
- `CONTAINER_REGISTRY`: Container registry host address
- `KUBE_CONFIG_BASE64`: Base64 encoded kubeconfig to connect to the k8s cluster

#### Automation Steps

1. Clone the repo and its sub-repos using `actions/checkout@master`.
2. Setup the `.env` with a random image tag
3. Build and push the production image using the generated tag
4. Setup Helm and kubectl using the Azure actions `azure/setup-helm@v3`, `azure/setup-kubectl@v3`
5. Modify `values.yaml` with our new configuration and update the Helm installation

> Opportunities from here are limitless; you can easily have any pull to the main deploy a feature environment or connect a backend to your Next.js app.

Simple steps to build, push, and deploy a Next.js app on Kubernetes

Quickly Deploy Any Next Js App On Kubernetes


Deploying Postgres on Kubernetes need not be complex. With automation in place, you can deploy a secure and isolated instance of PostgreSQL within your Microk8s cluster using Bitnami's Helm chart. The updated script covers everything from provisioning TLS certificates to configuring the database for secure connections.

### Preparing the Environment

The process starts with some initial setup tasks. We create a Kubernetes namespace for the PostgreSQL instance and establish the baseline for our commands:

```bash
#!/bin/bash

KUBECMD_PREFIX="${KUBECMD_PREFIX:=microk8s}"
CERT_MANAGER_NAMESPACE="cert-manager"

$KUBECMD_PREFIX kubectl create namespace $K8_NAMESPACE || true
```

### Configuring SSL/TLS Encryption

With cert-manager installed in our cluster, we can automatically provision and manage TLS certificates. This is a crucial step in ensuring that our database communication is encrypted:

```bash
cat <<EOF | $KUBECMD_PREFIX kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: $RELEASE_NAME-postgresql-tls
  namespace: $K8_NAMESPACE
spec:
  secretName: $RELEASE_NAME-postgresql-tls
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod
  dnsNames:
  - $DB_HOSTNAME
EOF
```

It's imperative to wait for the certificate to become ready. This script assures that by incorporating a timeout for the process:

```bash
$KUBECMD_PREFIX kubectl -n $K8_NAMESPACE wait --for=condition=ready certificate $RELEASE_NAME-postgresql-tls --timeout=300s
```

### Installing PostgreSQL with Helm

Having confirmed that the TLS certificate is in place, we proceed with the Helm chart installation:

```bash
$KUBECMD_PREFIX helm install $RELEASE_NAME oci://registry-1.docker.io/bitnamicharts/postgresql \
    -n $K8_NAMESPACE \
    --set global.postgresql.auth.username="$DB_USERNAME" \
    --set global.postgresql.auth.password="$DB_PASSWORD" \
    ...
    --set tls.certificatesSecret="$RELEASE_NAME-postgresql-tls" \
    --set tls.certFilename="tls.crt" \
    --set tls.certKeyFilename="tls.key"
```

The script passes the appropriate values to Helm, signaling it to deploy Postgres with TLS enabled.

### Networking Configuration

The `ConfigMap` and `Service` definitions are applied to route external traffic to the Postgres instance:

```bash
read -r -d '' INGRESS_CONFIG << EOM
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: ingress
  name: nginx-ingress-tcp-microk8s-conf
data:
  5432: "$K8_NAMESPACE/$RELEASE_NAME-postgresql:5432"
---
apiVersion: v1
kind: Service
metadata:
  name: $RELEASE_NAME-nodeport
  namespace: $K8_NAMESPACE
spec:
  type: NodePort
  ...
  nodePort: $TARGET_PORT
  ...
EOM

$KUBECMD_PREFIX kubectl apply -f - <<< "$INGRESS_CONFIG"
```

### Testing the Database Connection

Once deployed, verify the secure connection using Python and the `psycopg2` library. The script aims to establish a connection, execute a command, and then gracefully terminate the connection:

```python
import psycopg2

conn = None
try:
    conn = psycopg2.connect(
        dbname='...',
        user='...',
        password='...',
        host='...',
        port='...',
        sslmode='require'
    )

    print("Success: Connected to the database!")
    cur = conn.cursor()
    cur.execute("SELECT version();")
    record = cur.fetchone()
    print("You are connected to - ", record, "\n")
    cur.close()
except (Exception, psycopg2.DatabaseError) as error:
    print("Error: ", error)
finally:
    if conn is not None:
        conn.close()
        print("Database connection closed.")
```

By requiring `sslmode='require'`, the script confirms that the connection is not only secure but also compliant with best practices.

### Wrapping Up

The combination of Microk8s's simplicity and Helm's power simplifies the deployment of a robust PostgreSQL database. TLS encryption, managed by cert-manager, makes sure the data remains secure in transit. For developers and administrators looking to setup PostgreSQL on Kubernetes, the process is now streamlined, automated, and immeasurably more secure than before.

Do check out the full deployment script and accompanying resources on the [GitHub repository](https://github.com/tbscode/tims-blog-posts/blob/main/assets/postgress_create_dynamic_k8s.sh) and ensure your databases are secure from setup to daily operations.

This guide provides a step-by-step approach to installing a PostgreSQL database using Bitnami's Helm chart on a Microk8s cluster, including TLS configuration for enhanced security.

Setting Up a Secure Bitnami PostgreSQL Instance on Microk8s


Every now and then, when setting up a simple staging environment, the need for a quick-setup database that retains state consistently arises. In such situations, I often lean towards MariaDB, specifically utilizing the [Bitnami Helm Chart](https://github.com/bitnami/charts/tree/main/bitnami/mariadb), with slight tweaks to achieve the following:

- An auto-set, namespace setup
- Accessibility through a single host subdomain + port
- Easy hourly backups to a specified host path
- Exposure through TCP to the cluster host

### Prerequisites

This post is operating under the assumption that you have a `microk8s` Kubernetes cluster already set up, complete with `cert-manager`, `ingress`, `dns` and an active and configured `letsencrypt-prod` cluster issuer.

> Note: To jumpstart your setup, you might find [this Blog Post on Microk8s Private Cluster Setup](/blog/microk8s-on-vps) quite helpful.

### TL;DR 

For a swift deployment of this configuration:

1. [Grab this script](https://github.com/tbscode/tims-blog-posts/blob/main/assets/create_mariadb.sh)
2. Then run it as follows:

```bash
./create_mariadb.sh \
  RELEASE_NAME="<release-name>" \
  K8_NAMESPACE="<installation-namespace>" \
  DB_USERNAME="<db-username>" \
  DB_PASSWORD="<your-super-secure-password>" \
  DB_NAME="<db-name>" \
  TARGET_PORT="<desired-port>" \
  HOST_BACKUP_PATH="<your host backup volume>"
```


> Remember to expose the desired port through your firewall with a `sudo ufw allow $TARGET_PORT` command.
  
#### Uninstallation Process:

Execute the following command: 

```
microk8s helm uninstall $RELEASE_NAME
```

#### Connecting to the Database

Use this command to establish a connection to the database:

```bash
docker run -it --rm  mysql mysql --host <your-host-domain> --port <your-port> --user <your-user> --password --protocol=TCP
```

Or use this with `--network="host"` if your testing a local server.
Remember to enter the admin password when prompted.

#### Backing up the Database

Here's a simple docker command for accomplishing this:

```bash
docker run --network="host" --rm mysql:5.7 mysqldump -h localhost -P 30004 -u <your-user> -p<your-admin-password> --protocol=TCP <your-db-name> > backup.sql
```

#### Host Backup Strategy

A straightforward example of a cron job that allows your database to be backed up regularly is created as follows:

1. First, create a secret to hold your MariaDB credentials:

```bash
microk8s kubectl create secret generic mariadb-creds --from-literal=password='$DB_PASSWORD' --from-literal=username='$DB_USER' -n $K8_NAMESPACE
```

2. Then, define a CronJob in a yaml file:

```yaml
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: backup-job
spec:
  schedule: "0 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          volumes:
            - name: backup-volume
              hostPath:
                path: "$HOST_BACKUP_PATH"
          containers:
            - name: backup-container
              image: mysql:5.7
              volumeMounts:
                - name: backup-volume
                  mountPath: /backup
              env:
                - name: MYSQL_PWD
                  valueFrom:
                    secretKeyRef:
                      name: mariadb-creds
                      key: password
                - name: MYSQL_USER
                  valueFrom:
                    secretKeyRef:
                      name: mariadb-creds
                      key: username
              command: ["sh", "-c", 'mysqldump -h database-host -P 30004 -u $MYSQL_USER --protocol=TCP database-name > /backup/backup-$(date +%Y%m%d-%H%M%S).sql']
          restartPolicy: OnFailure
```

With this configuration in place, you would have a CronJob that executes hourly backups of the database.



This straightforward guide will walk you through deploying and exposing a basic MariaDB database on your microk8s Kubernetes cluster.

MariaDB on Kubernetes with Backup Cron-job


> Are you tired of overpriced package registries? Rapidly deploy your own git and package registry - with [Gitea](https://about.gitea.com/) - using [their helm chart](https://gitea.com/gitea/helm-chart).

### Prerequisites

This post assumes that you have a `microk8s` cluster set up with `cert-manager`, `ingress`, and `dns` configured, and a cluster issuer `letsencrypt-prod` ready.

> To get started, feel free to follow my [Blog Post on Microk8s Private Cluster Setup](/blog/microk8s-on-vps).

## Setting it Up

> **Please note** - This is meant for a quick temporary private package. There is **no persistence intended.**

For easy installation, you can use [this simple script](https://github.com/tbscode/tims-blog-posts/blob/main/assets/install_gitea_microk8s.sh)

```bash
./install_gitea_microk8s.sh \
  K8_NAMESPACE="<installation-namespace>" \
  RELEASE_NAME="<release-name>" \
  ADMIN_USERNAME="<...>" \
  ADMIN_PASSWORD="<...>" \ 
  INGRESS_HOST="<host-url>"
```

In the above script, `$INGRESS_HOST` represents a subdomain you've configured a DNS entry for, pointing to your server, for instance: `my-git.example.com`.

### Configuration

First, to ensure that this doesn't consume too many resources, we disable the high-availability database and use the default one instead.

```bash
microk8s helm install $RELEASE_NAME gitea-charts/gitea \
    -n $K8_NAMESPACE \
    --set postgresql-ha.enabled=false \
    --set postgresql.enabled=true \
```

Next, we establish some credentials for our default admin user:

```bash
    --set gitea.admin.username="$ADMIN_USERNAME" \
    --set gitea.admin.password="$ADMIN_PASSWORD" \
```

We then enable and configure the ingress. Please note we are increasing the `proxy-body-size` to 1GB to allow us to push larger packages.
```bash
    --set ingress.enabled=true \
    --set "ingress.annotations.kubernetes\.io/ingress\.class=public" \
    --set "ingress.annotations.cert-manager\.io/cluster-issuer=letsencrypt-prod" \
    --set "ingress.annotations.nginx\.ingress\.kubernetes\.io/proxy-body-size=1g" \
    --set ingress.hosts[0].host="$INGRESS_HOST" \
    --set ingress.hosts[0].paths[0].path=/ \
    --set ingress.hosts[0].paths[0].pathType=Prefix \
    --set ingress.tls[0].secretName="git.$RELEASE_NAME-tls" \
    --set ingress.tls[0].hosts[0]="$INGRESS_HOST" \
```

> To make packages of the admin user private, sign in at `$INGRESS_HOST`, navigate to Settings > Profile > Visibility, and set it to Private.

### Uploading a Package

You can push a package to a path under your Gitea user as follows:

```bash
echo "<your-admin-password" | docker login $INGRESS_HOST -u <your-gitea-admin> --password-stdin
docker tag <image-id> $INGRESS_HOST/<your-gitea-admin>/<some-package-name>
```
### Authenticating Deployments for Pulling Images

It's essential to authorize your deployments to pull your private container images. You can create a simple image pull secret in your Helm chart as outlined below.

```yaml
kind: Secret
type: kubernetes.io/dockerconfigjson
apiVersion: v1
metadata:
  name: dockerconfigjson-github-com
  namespace: {{ .Values.rootNamespace }}
stringData:
  .dockerconfigjson: >
    {{
      (
        dict "auths"
        (
          dict "$INGRESS_HOST"
          (
            dict "auth" .Values.registryAuth.token
          )
        )
      )
      |
      toJson
    }}
{{- end }}
```

> If you're using Github packages, for instance, `$INGRESS_HOST` would be `ghcr.io`

Create a token by Base64 encoding the `user:password` string:

```bash
echo "<gitea-admin>:<gitea-admin-password" | base64
```
This will allow you to pull from your private repositories.

### Setting up gitea action runners via helm



This guide will demonstrate how you can effortlessly host Gitea on a private Kubernetes cluster and utilize it as a package registry.

Self-Hosted Git Server with Gitea on Kubernetes


> NOTE: There is an [updated version of this post here](https://blog.t1m.me/blog/microk8s-on-vps-v2)
> Discover the cheapest and easiest way I've found to create personal Kubernetes clusters.

This blog post provides a step-by-step guide to setting up a private Kubernetes cluster on a Virtual Private Server (VPS), with the following features ready and configured:

- VPS firewall setup to expose HTTP/HTTPS & kubeapi server
- MetalLB configured for load balancing and exposing VPS IP
- Ingress and cert-manager setup for quick & automatic TLS management
- Kubeapi server configured for restricted remote access

The **prerequisites** for this tutorial are: **A VPS with a Public IP**, **A domain with a configurable DNS**.

## Step 1: Create a base user

Connect to your VPS. In a terminal, create a user and add them to the sudoers.

```bash
adduser <user-name>
usermod -aG sudo <user-name>
```

## Step 2: Create SSH keys

On your local machine, create SSH keys to access the server.
Set a secure password for the key.

```bash
ssh-keygen -b 2048 -t rsa
chmod 600 <key-file> <key-file>.pub
```

Copy the content of `<key-name>.pub` to your VPS at `/home/<user-name>/.ssh/authorized_keys`.

Now we edit the ssh config to require the ssh key, edit `/etc/ssh/sshd_config`

```bash
PasswordAuthentication no # yes before
PubkeyAuthentication yes # no before
```

Then restart the ssh service `service ssh restart`

> Note: these are minimal ssh setup setups and depending on the environment extra steps should be taken to make ssh more robust agains outside attacks.

## Step 3: Set up and enable the firewall

Set up the firewall to allow SSH, HTTP, HTTPS connections and also expose port `16443` which will be used to access the Kubeapi server.

```bash
ufw allow ssh
ufw allow http
ufw allow https
ufw allow 16443
ufw enable
```

Now disconnect from your server and reconnect as the new user:

```bash
ssh -i "<private-server-key>" <user-name>@<server-IP>
```

## Step 4: Install and set up microk8s

```bash
sudo snap install microk8s --classic --channel=1.28
sudo microk8s enable rbac
sudo microk8s start
microk8s enable ingress dns hostpath-storage cert-manager
```

If you want to use the current user to manage microk8s; this is not recomended for production clusters

```
sudo usermod -a -G microk8s <your-user>
```

Now we create a default cluster issuer: `letsencrypt-prod`:

```bash
microk8s kubectl apply -f - <<EOF
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-prod
spec:
 acme:
   email: <your email>
   server: https://acme-v02.api.letsencrypt.org/directory
   privateKeySecretRef:
     name: lets-encrypt-priviate-key
   solvers:
   - http01:
       ingress:
         class: public
EOF
```

We also need to enable some additional rules for the firewall ([see this github comment](https://github.com/canonical/microk8s/issues/2418#issuecomment-877350375)):

```bash
sudo ufw allow in on cali+
sudo ufw allow out on cali+
```

To check for general configration issues use `microk8s inspect`
This can give you some easy setup configuration infos and warnings such as:

To enable ip forwarding:

```
sudo iptables -P FORWARD ACCEPT
sudo apt-get install iptables-persistent
```


No to konfigure kubeserver access through the host dns, edit `/var/snap/microk8s/current/certs/csr.conf.template`:

```bash
[ alt_names ]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster
DNS.5 = kubernetes.default.svc.cluster.local
DNS.6 = k8s.<your-domain>
```

Refresh the server certificate:

```bash
sudo microk8s refresh-certs --cert server.crt
```

View the updated client certificate: `microk8s config > kubeconfig.yaml`.
Open the `kubeconfig.yaml` and edit the `server:` entry.

```bash
...
- cluster:
    certificate-authority-data: XXXXXXXXCENSOREDXXXXXXX
    server: <pub-IP> <---- put k8s.<domain> here
...
```

Now you can connect to the Kubernetes cluster from your local machine:

```bash
KUBECONFIG="./kubeconfig.yaml" kubectl get pods --all-namespaces
```

## Step 5: Setting Up MetalLB

First, prepare `<your-domain>`. 
Create a specific or one wildcard DNS entry:

```bash
<your-domain> -> <vps-ip>
*.<your-domain> -> <vps-ip>
```

Enable `metallb` via `microk8s enable metallb:<your-ip>-<your-ip>`.

To make `metallb` work correctly, apply the patch described in [this GitHub comment](https://github.com/canonical/microk8s/issues/824#issuecomment-1003284063).


```bash
#!/bin/sh

command -v kubectl > /dev/null 2>&1 && KUBECTL=kubectl
command -v microk8s.kubectl > /dev/null 2>&1 && KUBECTL=microk8s.kubectl

INGTMPFILE=$(mktemp -t ingress_daemonset.XXXXXXXX)

trap "rm -f ${INGTMPFILE}" 0 1 2 3

${KUBECTL} -n ingress get daemonset nginx-ingress-microk8s-controller -o yaml | \
    sed -e 's|- --publish-status-address=.*|- --publish-service=$(POD_NAMESPACE)/ingress|' > ${INGTMPFILE}

${KUBECTL} diff -f ${INGTMPFILE}
if [ $? -eq 0 ]; then
    echo "No changes need to be made"
else
    ${KUBECTL} apply -f ${INGTMPFILE}
fi
```

You can now use LoadBalancer nodes that distribute the `<VPS-IP>`.

## Some Tips

#### Enabling tab completion for microk8s

`vim ~/.bash_aliases`

```bash
# add the fllowing
alias kubectl='microk8s kubectl'
export LC_ALL=en_US.utf-8
export LANG=en_US.utf-8
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)  
```

#### Patching External Traffic Policies

```bash
kubectl patch svc nodeport -p '{"spec":{"externalTrafficPolicy":"Local"}}'
```

#### Testing the preservation of client IP adresses

```bash
kubectl expose deployment source-ip-app --name=loadbalancer --port=80 --target-port=8080 --type=LoadBalancer
```

## Conclusion

You now have a Kubernetes cluster with services that can be exposed via ingress, have a load balancer, and cert manager ready. You should be able to access these services through a host domain. 

> The possibilities are endless from here ;^)

You even have the option to connect this to multiple VPSs to extend your cluster. More on that in a future blog post.


A step-by-step guide to setting up a fully configurable private Kubernetes cluster

Setting up a Private Kubernetes Cluster on Your Own VPS Using Microk8s


> UPDATE: We made 2. Place! 🥳 Checkout [the hackathon page](https://devpost.com/software/tim-s-stack-anystack).

<iframe width="560" height="315" src="https://www.youtube.com/embed/_06vvSltvvY?si=ZMTgD9l2TFNTTa_2" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

> Checkout the [full stack code](https://github.com/tbscode/tims-stack-anystack) on my repo

## Introduction

I recently participated in the Bunnyshell hackathon, where I successfully orchestrated a dynamic cross-platform web app stack. I want to share with you in detail my development process, core features, and the benefits of this stack.

This stack is primarily designed for dynamic, real-time web apps, with a specific inclination towards mobile clients. One of the prominent features is that any changes made in the backend can be directly and instantly transmitted to the clients via a WebSocket connection, thanks to Redux for smooth state management.

## Architecture

I used a Docker container-based architecture using Helm for my application. Docker simplifies the process of managing my application's lifecycle, while Helm assists in deploying my Docker images into a Kubernetes cluster.

Here is an image representation of my stack's architecture:

![stack overview](/static/assets/overview_graph.png)

## Building the Stack

The stack incorporates:

- **Django Backend**: Django is utilised for the backend due to its simplicity and rapid development capabilities. Features like Celery for task management and DRF Spectacular for API documentation are added advantages.

- **Next.js + React Frontend**: Next.js powers the frontend, providing performance benefits, SEO capabilities, and better developer experience.

- **Postgresql & Redis**: They act as primary databases for data and celery respectively.

- **Documentation**: Pdoc3 was used to generate code documentation from backend code.

## Configuring the Stack on Bunnyshell

Bunnyshell significantly simplifies cloud management and deployment automation making it a breeze to set up a Kubernetes cluster and to deploy my application there.

In addition to this, I also exemplify setting up a private Microk8s cluster. Microk8s serves as a lightweight, lean, and efficient Kubernetes distribution, enabling quick and easy deployment of Kubernetes resources even on a low-end VPS.

I hope you find this stack setup helpful and efficient for your next big project. The GitHub repository contains all the code with appropriate comments and READMEs for further exploration.

Happy Coding!

## Technologies

The following table provides a quick overview of the key technologies integrated into the stack:

| Component | Technology | Purpose |
|-----------|------------|---------|
| Frontend  | Next.js + React | Rich UI |
|           | Tailwind CSS + DaisyUI | Styling |
|           | Redux | State Management |
|           | Capacitor | Native integrations and iOS/Android PWA export |
| Backend   | Django | Application Framework |
|           | Celery | Task Management |
|           | Django REST Framework + django_rest_dataclasses | REST API Development |
|           | DRF Spectacular | API Documentation |
|           | Django Channels | Managing WebSockets |
| Documentation | Pdoc3 | Code Documentation |
| Database  | PostgreSQL | Primary Backend Database |
|           | Redis | Broker for Celery and Django Channels |
| Containerization | Docker | Container Creation and Management |
| Deployment and Orchestration | Helm + Bunnyshell | Simplified deployment, Scaling and Management |

Each of these technologies has been selected for its specific capabilities that contribute to the efficient operation of the entire web application stack.

This article showcases a high-performing web app stack I created for a recent Bunnyshell hackathon, using Docker, Kubernetes, Microk8s, Django and Next.js.

Tim's Stack: Dynamic Cross-platform Web App Stack, deployed with Bunnyshell


Art Collective Simulator is a Game Dev Story–inspired simulation for the Vibrational Network, a Hamburg-based collective of musical and visual artists. I developed the codebase, while Giaco and Togrul contributed graphics and music.

The project evolved rapidly across several months, adding systems for characters, tasks, rooms, and events, alongside a modernized dynamic UI and mobile support.

### Feature timeline

#### Up to Jan 2020

- Task section
- Stat-based score generation
- Personalizable and random characters
- First event-related states (e.g., playing instruments)
- Event guest simulations
- Multiple rooms and teleports
- Completely updated dynamic UI

![Jam session gameplay](/assets/archive/game_play_new_jamsession.gif)

#### July

- Full game-state saving and loading
- Game phases
- Phase-based task selection
- Automatic task-based score generation
- Several new game tasks
- Android and iOS compatibility

![June/July gameplay](/assets/archive/june_july.gif)

#### June

- A* pathfinding
- Character stats such as Intelligence, Speed, Creativity
- Stats-based scoring system

#### May

Started programming the simulations with a complete task system. Tasks control entities by working through task queues. Tasks are selected based on player actions and game state.

![Simulation](/assets/archive/simulation.gif)

#### April

First visual prototype: basic isometric functionality — y-sorting, 2D movement, clickable tiles, character animation, and walk cycles.

![First gameplay](/assets/archive/first_game_play.gif)

#### March

I started developing the general framework using the Godot game engine. I outlined core concepts and scoped the project. The following diagrams show interactions between scores, entities, and events — the game's main components.

![Implementation](/assets/archive/implementation.jpg)

![Event hosting](/assets/archive/event_hosting.jpg)

#### February

I began the collaboration with the Vibrational Network in Hamburg — a collective of musical and visual artists. A friend reached out about developing a game; I was motivated to build my first isometric game: a Game Dev Story–style simulation with art.




A Game Dev Story–inspired isometric simulation for the Vibrational Network with evolving features from tasks and stats to events and rooms.

Art Collective Simulator


Retro Escape is a simple but challenging puzzle game. Guide a character from the start to the end tile while navigating walls, collecting items in the right order, and using them appropriately to solve each level.

However, that's not all — programmable blocks and recursive objects unlock new puzzle concepts and level designs.

Not only can you challenge your mind by solving tricky levels, you can also use your creativity to build unique ones. The built-in level editor includes all game blocks and offers a simple script language to interface with the programmable blocks. Created levels can be saved or shared via file and QR code.

The game was developed cross-platform using the LibGDX framework. It was written in Kotlin and uses the ZXING library for QR code detection. An Android beta was released featuring 30 levels.

### Gallery

![Retro Escape banner](/assets/archive/banner_rtesc_3.png)

![Retro Escape banner](/assets/archive/banner_rtesc_4.png)

![Retro Escape banner](/assets/archive/banner_rtesc_5.png)

![Retro Escape banner](/assets/archive/banner_rtesc_6.png)

![Gameplay](/assets/archive/retro_escape_play.gif)




A simple yet challenging puzzle game with programmable blocks, a level editor, and QR-based sharing. Built with Kotlin/LibGDX and ZXing.

Retro Escape


Frequency Quick Response (FQR) Codes enable sending small to medium-sized files by converting data into a sequence of QR codes.

This sequence is displayed by an app on the sender's device and captured by the receiver's camera — no network connection required.

Because transfer happens via QR codes, an FQR sequence can also be printed, passed on paper, and re-scanned later. For example, small flipbooks can be used as physical data carriers and distributed as a novel brochure format.

One major advantage of FQR is that it does not require a connection like Wi‑Fi, Bluetooth, or cellular, which might incur costs or be unavailable. In areas without mobile coverage — or when large files would deplete data plans — FQR can be a faster and more practical alternative on smartphones.

Like traditional QR codes on paper, FQR codes can be shown on screens — in broadcasts or shop windows — to transfer additional materials directly to users' devices.

### Further reading

- [Ausarbeitung (PDF)](/assets/archive/FQRAusarbeitung.pdf)
- Jugend forscht: Frequency Quick Response Code: https://www.jugend-forscht.de/projektdatenbank/frequency-quick-response-code.html
- Weser Kurier: Erfolgreiche Forscher: https://www.weser-kurier.de/region/die-norddeutsche_artikel,-erfolgreiche-forscher-_arid,1587114.html
- „Jugend forscht“: Ideen aus dem Klassenzimmer: https://www.kreiszeitung.de/lokales/bremen/ideen-klassenzimmer-8008471.html

![FQR infographic](/assets/archive/fqr_info_graphic.png)




Offline file transfer by streaming sequences of QR codes — printable, scannable, and network-free.

Frequency Quick Response (FQR) Code


This list is in no way exhaustive, nor does it deeply analyze and or compare.
The list is meant for later reference and comments, if you find factually false information, please contact me `tim+blog@timschupp.de`.
If you have comments are discussion points, also contact me. For a [change log, check the github history of this article](https://github.com/tbscode/tims-blog-posts/).

The developments in Surveillance are related to the power exercises of a few,
there are many more laws and regulation being instated that don't directly foster surveillance but ensure retention of power and ongoing corruption in governments and state actors. This goes so far that we pass many laws that have 'seemingly good intentions' but contain many causes that directly consolidate power of a few companies and actors, **this is a whole topic in itself and thus will receive it's own blog article in the future**.

<p style="text-align:center;">
  <img src="/static/assets/ClippyTimDark.png" alt="Clippy and Calvin" />
</p>

#### Why this list?

I'm convinced that misinformation and surveillance are some of the biggest Threats to democracy our way to live and our freedom to manage and control the live we lead. Especially currently I feel that a lot of the privacy protections the EU had in place are weakening and we are rapidly making steps towards surveillance.
Therefore I created this curated list, so I can share it when this topic is touched in a conversation.

Some quotes:

- `"[NSA] surveillance… threatens the core principles of a democratic society."` — Sir Tim Berners-Lee (2013)
- `"Spying on individuals on a massive scale, without strict legal rules and democratic oversight, ... harms freedom of speech, association and participation."` — Council of Europe Commissioner for Human Rights (2013)

## `#UK-1` Online Safety Act

Under the umbrella of 'protect children online' the UK-Online-Safety-Act enables nationwide censorship of content.
It enforces age-checks on any publicly accessible web-page in the UK that host content which is 'age-restricted'.
The UK's Office for Communications (Ofcom) is enforcing the law by fining website owners that don't introduce age verification.
Age verification 

Additional Concerns are that the Online Safety act also contains status around *Encryption & client-side scanning*,
which some argue could enable even further reaching surveillance.

**Tim's Comment**: The enforcement hits the wrong target here. If it were actually fulfilling the goal of online safety, it would be no issue at all to enable global authentication at carrier level, thus restrict children's access without enforcing change with the platform owners. The way the law is currently set-up it allows for targeted censorship, which use we have already seen (3). Also forcing plattform owner to perform the age verification on their plattform but trough a 3rd party ventor, opens up many new angels for internet usage tracking of individuals.
The EU's Digital Services Act is a somewhat 'weaker' but well better designed law that enforce similar but more controlled measures to protect against similar dangers.

#### Reference:

1. [Wikipedia](https://en.wikipedia.org/wiki/Online_Safety_Act_2023)
2. [Discord Data Breach](https://proton.me/blog/discord-age-verfication-breach) Though this is only indirectly related to the safety act it clearly shows the danger of processing and storing ID's.
3. [Safety Act Used For Censorship Instead Protection](https://www.theguardian.com/technology/2025/aug/04/social-media-battles-and-barbs-on-both-sides-of-atlantic-over-uk-online-safety-act?utm_source=chatgpt.com)


## `#EU-1` Chat-Control

Since 2022 brought to the EU courts multiple times, a law under the name "Chat-Control" Proposes measure toward ensuring child safety, combating [CSAM](https://en.wikipedia.org/wiki/Child_pornography) online, targeting specifically messaging services.
 
The proposal calls for scanning messaging content for CSAM or abuse, many experts have noted that this crashes with end-to-end-encryption on a technical level, and thus would only be feasible with client side scanning, which in turn bypasses the whole concept of end to end encryption.

**Tim's Comment**: Again, enforcement seems misdirected. If the actual goal is protecting children, this can be pursued with targeted, court-supervised investigations and better cross-border policing **and especially better education and measure for parents to protect their kids**. But not through normalizing client side bulk scanning. Mandating mechanisms that break or bypass E2EE opens the door to selective filtering and abuse. As with many other these, we know **existing capabilities WILL be abused**.

#### Reference:

1. [Wikipedia: EU regulation on CSAM detection/"Chat Control"](https://en.wikipedia.org/wiki/Chat_control)
2. [EDRi: Why client-side scanning is a threat to privacy and security](https://edri.org/our-work/chat-control-what-is-actually-going-on/)
3. [Signal: Scanning content on your phone breaks end-to-end encryption](https://aboutsignal.com/news/the-end-of-private-conversations-signal-threatens-to-leave-the-eu-if-chat-control-becomes-mandatory/).
4. [Bugs in our Pockets: The Risks of Client-Side Scanning](https://arxiv.org/abs/2110.07450)


## `#EU-2` Facial Scanning and the EU-AI-Act

The new EU-AI-Act contain several regulation in terms of use and application of facial recognition - which can be in-part seen as positive protection - of application for facial scanning, and the use of biometric data.
If compared to several other countries this law is very careful and mandates regulatory oversight and allows use only in narrowly defined scenarios, but regardless it mandates the use of facial scanning in several scenarios. It gives the regulatory decision power to the member state in question and enforces reporting and self-regulation.

**Tim's Comment**: Forbidding facial scanning in many scenarios is a step in the right direction! But the effectiveness and the bias of regulatory oversight for the cases where it will be permitted is to be questioned. Also the member state enforcement or actions if these rules are broken aren't defined well. Thus generally more options for surveillance are given and roll-out of these systems will be supported by these regulation existing.


#### Reference:

1. [EU Parliament](https://www.europarl.europa.eu/news/en/press-room/20240308IPR19015/artificial-intelligence-act-meps-adopt-landmark-law#:~:text=Law%20enforcement%20exemptions%20The%20use,linked%20to%20a%20criminal%20offence)

## `#GOGL-1` Mandatory developer verification coming for all APP's

Starting in September 2026 (with rollout stages beginning March 2026), Google will require **developer verification** for **every app** installed on certified Android devices—including apps from third‑party stores and direct APK installs; ADB installs are the notable exception (1)(2)(4). Any install outside Play (F‑Droid, other stores, direct downloads) must be tied to a verified legal identity and registered package names, extending Play‑style checks to off‑Play distribution (1)(3).

Practically, anonymous or hobbyist distribution becomes much harder: unverified installs ( which they want use to call 'sideloads' ) will fail on certified devices, and third‑party stores can only ship apps from verified developers (4)(5). Google presents this as curbing PHAs and ban‑evasion while "keeping Android open and safe" (1).

This coincides with broader use of Play Integrity, where apps check "installed by Play" and device/strong integrity. That tends to exclude non‑certified OSes like GrapheneOS despite strong security properties, leading to breakage in banking, payments, transit, streaming, and games when developers enforce those verdicts (5)(6).

**Tim's Comment**: Useful against re‑uploads, but it centralizes control and blurs certification with security. Prefer **hardware key attestation** for stronger, targeted trust—and treat "installed by Play" and device/strong integrity as advisory, not hard gates, so secure alt‑OS users aren’t collateral (6)(5).

#### Reference:

1. [Android Developer Verification Policy](https://developer.android.com/developer-verification?utm_source=chatgpt.com)
2. [Android Developers Blog: Rollout Timeline](https://android-developers.googleblog.com/2025/08/elevating-android-security.html?utm_source=chatgpt.com)
3. [The Verge: Google Verification Requirement](https://www.theverge.com/news/765881/google-android-apps-side-loading-developer-verification?utm_source=chatgpt.com)
4. [Hackaday: Non-ADB APK Installs Require Registration](https://hackaday.com/2025/10/06/google-confirms-non-adb-apk-installs-will-require-developer-registration/?utm_source=chatgpt.com)
5. [Play Integrity API Overview](https://developer.android.com/google/play/integrity/overview?utm_source=chatgpt.com)
6. [GrapheneOS: Attestation Compatibility Guide](https://grapheneos.org/articles/attestation-compatibility-guide?utm_source=chatgpt.com)
7. [F-Droids statement](https://f-droid.org/en/2025/09/29/google-developer-registration-decree.html)


## `#MSFT-1` Windows Recal + MSA tied TUID

Windows 11 develops in a similar way as Android and IOS with their device integrity apis.
With Win11 TPM 2.0 and Secure Boot are enableed by default and and siginifantly harder up to impossible to circumvent.
This plus moving windows towards being usable with Microsoft Account, Any attestation, any system callback, anything that leaves a trace really, can easily be linked to your microsoft identity. And even worse with the Mandatory Roll-Out of Windows-Recall, which is supposed to take screenshots every 5 minutes, even apps that try to protect your privacy will be exposed. All Security Prommises microsoft makes are directly hiding a pricacy concession, microsoft also joins the companies normalizing hardware-rooted, cloud tied attestation.

TPM 2.0 + Secure Boot are the default for supported installs, while setup heavily steers you into a Microsoft Account. The TPM isn't "spyware"; it's a secure coprocessor (keys, PCR measurements, sealing, attestation). But combined with Windows' security stack (BitLocker/Device Encryption, VBS/HVCI, Windows Hello) and the cloud defaults (OneDrive/Store/Widgets/Copilot), it makes proving "this exact machine, in this exact state, used by this signed‑in person" easy—when services choose to enforce it. That's great for stopping disk theft and some malware; it's bad for staying low‑friction anonymous. Remote attestation can become a gate: prove genuine Windows and an unmodified boot chain or get blocked, a model already used in enterprise and increasingly mirrored by anti‑cheat/DRM‑like systems. With MSA, devices are registered, encryption keys may be backed up, and telemetry/online features link behavior to account and hardware; Pluton on some CPUs further centralizes the root of trust. You can still go local‑account, trim telemetry, avoid MS services, use a VM, or switch OS—but these are now "advanced user paths," and the trend is to narrow them.

**Tim's Comment**: The mechanisms (TPM, Secure Boot) are genuinely useful for security, but the risk is how they're combined with mandatory online identity, opaque telemetry, and attestation used as an access/lockdown weapon. This shifts power from the device owner to the platform vendor. Remote attestation can become "show me you're running unmodified, approved software or you're out"—already happening in enterprise and anti‑cheat scenarios. The MSA requirement + TPM‑backed identity + telemetry systematicly threaten user anonymity and privacy. As with other surveillance capabilities, we know **existing capabilities WILL be abused**. The trend toward narrowing local‑account and offline paths, while pushing cloud‑first defaults, makes truly anonymous, de‑tethered usage more work than on older Windows—and Microsoft keeps closing workarounds.

#### References:

1. [The Hidden Spy in Windows 11 (TPM Chips)](https://www.youtube.com/watch?v=t1eX_vvAlUc)

### More to check-out

This Post isn't finished, nor does it contain really much, just contains what I was able to write down in a few hours.
I strongly encourage everyone to do their own research on the topic, circle back, comment, [and maybe also help me extend this post, simply open an issue on my github with your proposed changes.](https://github.com/tbscode/tims-blog-posts/issues/new/choose).

##### Reference:

1. [The Survailence Report](https://surveillancereport.tech/)
2. [Missfits: Enshitification](https://us.macmillan.com/books/9780374619329/enshittification/)
3. [Manufacturing Consent](https://en.wikipedia.org/wiki/Manufacturing_Consent) A book more relevant then ever. 
*Quick Call-out: 🔩 you Studien Stiftung Deutschland for discrediting 'Manufactoring Consent' as globally not relevant topic in 2017*
4. [Consumer Rights](https://consumerrights.wiki/w/Main_Page)
5. [Flock Safetly Cameras](https://www.youtube.com/watch?v=_a14lplBg2Y)

A quote from the Party in George Orwell's 1984. Indicators for increasing Surveillance, just a curated reference list, with some comments.

WordPress on Kubernetes with TrueCharts (short guide)

Chart

Example values

Keep Reading

Expose NAT Services Across Two Kubernetes Clusters with Tailscale

CloudNativePG PostgreSQL End-to-End Setup Guide

Building a private LLM Cluster