
Alexander is a writer, traveler, and serial digital entrepreneur. He’s the founder of the Tushi, the Gen X personal growth newsletter Further, and offers small business advice at Unemployable.

Twitter

Linkedin

Facebook

Director and Chief Evangelist

Alexander Hipp


Anil is a content-obsessed marketer who loves driving organic traffic for outstanding companies. He is also the creator of Hack the Content, co-host of The Runner, and owner of several digital commerce sites.

Managing Partner

Anil Vugels


Monika is a writer, musician, and content strategist with a passion for language. When she isn’t helping clients, you can find her writing songs, hiking in a northern forest, or furiously journaling.

Director of Operations

Ann Monika

Mastodon

Github

Programmer / Founder / Student

Tim Schupp


I use this pattern when I want to keep workloads private in a k3s cluster, but still expose selected services to the public internet through a separate public microk8s edge cluster.

This is a full, production-style walkthrough for exposing a private service across two clusters with Tailscale Kubernetes Operator.

It is tailored to this exact topology:

- Source (private) cluster: k3s, kubeconfig at `kubeconfig-node-A.yaml`
- Public edge cluster: microk8s, kubeconfig at `kubeconfig-node-B.yaml`
- Private app example: service `<app-service>` in k3s namespace `<app-namespace>`
- Public hostname example: `example.server.com`
- TLS issuer on public cluster: `letsencrypt-prod` using ACME HTTP-01

---

## 1. Architecture and Traffic Flow

Goal:

- Keep app workloads private in k3s.
- Publish them safely via public ingress in microk8s.
- Use Tailscale as encrypted inter-cluster transport.

Data path:

1. Client requests `https://example.server.com`.
2. DNS resolves to the public microk8s node IP.
3. microk8s ingress controller (`IngressClass=public`) receives request.
4. Ingress routes to `<app-service>-tailnet` service in microk8s.
5. Tailscale operator egress proxy in microk8s forwards traffic over tailnet.
6. Tailnet reaches the k3s Tailscale-exposed service `<app-hostname-node-a>.<tailnet>.ts.net`.
7. k3s service forwards to the private app pod.

This pattern works for many NAT/internal services, not only one specific app.

---

## 2. Prerequisites

### 2.1 DNS

- `example.server.com` must point to the public microk8s ingress IP.

### 2.2 cert-manager in public cluster

- `cert-manager` installed and healthy.
- `ClusterIssuer` named `letsencrypt-prod` configured for HTTP-01.

Check:

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl get pods -n cert-manager
kubectl get clusterissuer letsencrypt-prod
```

### 2.3 Ingress controller class in microk8s

In this environment, ingress class is `public` (not `nginx`).

Check:

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl get ingressclass
kubectl -n ingress get pods
```

### 2.4 Tailscale operator installed in both clusters

Check:

```bash
export KUBECONFIG=./kubeconfig-node-A.yaml
kubectl get pods -n tailscale

export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl get pods -n tailscale
```

Both should show the operator pod running.

---

## 3. Phase A: Expose Service in k3s (Source Cluster)

You can expose an existing `ClusterIP` service via annotations.

For service `<app-service>` in namespace `<app-namespace>`:

```bash
export KUBECONFIG=./kubeconfig-node-A.yaml
kubectl annotate service <app-service> -n <app-namespace> \
  tailscale.com/expose=true \
  tailscale.com/hostname=<app-hostname-node-a> \
  --overwrite
```

Verify:

```bash
export KUBECONFIG=./kubeconfig-node-A.yaml
kubectl -n <app-namespace> get svc <app-service> -o yaml
kubectl -n tailscale get pods,svc,secrets
```

Expected:

- The source service has `tailscale.com/expose: "true"`.
- A proxy pod appears in `tailscale` namespace (for example `ts-<app-service>-xxxxx-0`).
- A tailnet FQDN is created.

Get tailnet FQDN from secret:

```bash
export KUBECONFIG=./kubeconfig-node-A.yaml
kubectl -n tailscale get secret <proxy-secret-name> -o jsonpath='{.data.device_fqdn}' | base64 -d
```

Example value:

- `<app-hostname-node-a>.taild4f875.ts.net.`

Keep this FQDN. It is the inter-cluster target.

---

## 4. Phase B: Configure microk8s as Public Edge Cluster

Use Tailscale cluster egress mode in microk8s. This is the critical part.

Do not use a plain `ExternalName` directly to `<name>.ts.net` without Tailscale egress annotations. The ingress controller may fail DNS resolution and return `504`.

### 4.1 Manifest

Create `app-proxy.yaml` in the public cluster context with this content:

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: <app-namespace>
---
apiVersion: v1
kind: Service
metadata:
  name: <app-service>-tailnet
  namespace: <app-namespace>
  annotations:
    tailscale.com/tailnet-fqdn: <app-hostname-node-a>.taild4f875.ts.net
spec:
  type: ExternalName
  externalName: placeholder
  ports:
    - name: http
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: <app-service>-public
  namespace: <app-namespace>
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: public
  tls:
      - hosts:
        - example.server.com
      secretName: example-server-com-tls
  rules:
    - host: example.server.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: <app-service>-tailnet
                port:
                  number: 80
```

Apply:

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl apply -f app-proxy.yaml
```

Why `externalName: placeholder` is correct:

- Tailscale operator rewrites `spec.externalName` to an internal service it creates in `tailscale` namespace.
- This enables reliable egress proxying to tailnet targets.

---

## 5. Validation Workflow (End-to-End)

Run these checks in order.

### 5.1 In public cluster, check rewritten egress service

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n <app-namespace> get svc <app-service>-tailnet -o yaml
```

Expected:

- `spec.externalName` changed from `placeholder` to something like `ts-<app-service>-tailnet-xxxxx.tailscale.svc.cluster.local`.
- `status.conditions` includes `TailscaleProxyReady=True`.

### 5.2 Check tailscale proxy objects in public cluster

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n tailscale get pods,svc,secrets
```

Expected:

- pod `ts-<app-service>-tailnet-xxxxx-0` is `Running`.

### 5.3 Check ingress and certificate

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n <app-namespace> get ingress <app-service>-public -o wide
kubectl -n <app-namespace> get certificate,certificaterequest,order,challenge
```

Expected:

- Ingress class shown as `public`.
- Certificate `example-server-com-tls` is `Ready=True`.

### 5.4 In-cluster backend connectivity test

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n <app-namespace> run curlcheck --image=curlimages/curl:8.8.0 --restart=Never --rm -i --command -- sh -c "curl -sS -o /dev/null -w '%{http_code}\n' http://<app-service>-tailnet.<app-namespace>.svc.cluster.local"
```

Expected output:

- `200`

### 5.5 Public test

```bash
curl -I https://example.server.com
```

Expected:

- HTTP `200` (or app-specific expected code), valid TLS chain.

---

## 6. Troubleshooting by Symptom

### 6.1 Symptom: `504 Gateway Time-out`

Most common causes:

- Ingress points to wrong class (`nginx` instead of `public` in this microk8s setup).
- Service is direct `ExternalName -> <name>.ts.net` without `tailscale.com/tailnet-fqdn` egress annotation.
- Tailscale egress proxy pod not created or not ready.

Check ingress logs:

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n ingress logs <nginx-ingress-controller-pod> --tail=200
```

If you see repeated DNS `name error` for tailnet hostname or `placeholder`, your service wiring is incorrect or still converging.

### 6.2 Symptom: TLS cert not issued

Check:

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n <app-namespace> get certificate,certificaterequest,order,challenge
kubectl -n <app-namespace> describe ingress <app-service>-public
```

Common causes:

- `example.server.com` DNS not pointing to public ingress endpoint.
- Wrong `cluster-issuer` name.
- HTTP-01 solver path blocked by firewall/proxy.

### 6.3 Symptom: Ingress serves default cert

Cause:

- `spec.tls[].secretName` mismatched with cert-manager-produced secret.

Fix:

- Ensure `Ingress.spec.tls.secretName` equals certificate secret (for example `example-server-com-tls`).

### 6.4 Symptom: Tailscale operator ignores service

Check:

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n tailscale logs deploy/operator --tail=300
```

If logs show no egress reconciliation for your service, ensure:

- Service is type `ExternalName`.
- Has annotation `tailscale.com/tailnet-fqdn` or `tailscale.com/tailnet-ip`.
- Tailscale operator is healthy.

---

## 7. Operational Best Practices

- Use dedicated namespace per proxied application (for example `<app-namespace>`).
- Use explicit hostnames per service (for example `<app1-hostname-node-a>`, `<app2-hostname-node-a>`).
- Keep private cluster service as `ClusterIP` only; avoid public LB there.
- Terminate public TLS in edge cluster ingress.
- Monitor ingress and tailscale operator logs for DNS and proxy readiness conditions.
- Keep cert-manager issuer names consistent across manifests.

---

## 8. Pattern Reuse for Any NAT Service

To expose another private k3s service through the same public microk8s cluster:

1. Annotate source service in k3s with `tailscale.com/expose=true` and a unique hostname.
2. In microk8s, create annotated `ExternalName` service with `tailscale.com/tailnet-fqdn` to that hostname.
3. Create public ingress for the desired DNS host with `ingressClassName: public`.
4. Attach cert-manager annotation `cert-manager.io/cluster-issuer: letsencrypt-prod`.
5. Validate service rewrite, proxy pod readiness, and certificate readiness.

This gives a repeatable, encrypted cross-cluster edge publishing model.

---

## 9. Reference Commands Cheat Sheet

Source cluster (k3s):

```bash
export KUBECONFIG=./kubeconfig-node-A.yaml
kubectl -n <app-namespace> get svc <app-service> -o yaml
kubectl -n tailscale get pods,svc,secrets
```

Public cluster (microk8s):

```bash
export KUBECONFIG=./kubeconfig-node-B.yaml
kubectl -n <app-namespace> get svc,ingress
kubectl -n <app-namespace> get certificate,certificaterequest,order,challenge
kubectl -n tailscale get pods,svc,secrets
kubectl -n tailscale logs deploy/operator --tail=200
kubectl -n ingress logs <nginx-ingress-controller-pod> --tail=200
```

Public endpoint test:

```bash
curl -I https://example.server.com
```


A production-style setup to publish private k3s services through a public microk8s edge cluster using the Tailscale Kubernetes Operator.

Expose NAT Services Across Two Kubernetes Clusters with Tailscale


[Open-Chat](https://github.com/msgmate-io/open-chat-go) is not in it's 7th iteration and fully re-written in golang.
It now as 'federation' features using libp2p, it allows you to quickly spin-up your own network of open-chat nodes.

Open-Chat network member can freely communicate with another, they are automatically joined into a completely independent self-healing overlay-network.

### Creating a Open-Chat Network

Open-Chat Networks can be established inside private networks, across public networks *or across both*.
To be able to penetrate across public and private networks open-chat need to be able to route traffic back to itself trough a router, 
or it need to have at-least one public relay peer available to use as alternate path.

#### **1. Prepare a Bootstrap and Relay Node**
On our public node, we want to use as 'relay' and 'bootstrap' peer, download the Open-Chat binary [from the github release page](https://github.com/msgmate-io/open-chat-go/releases)
Be sure to download the one with '-federation' build in.
E.g.: 

```bash
wget -O open-chat https://github.com/msgmate-io/open-chat-go/releases/download/build-10/open-chat-federation-0.0.111-linux-amd64
chmod +x open-chat # make it executable
```

#### **2. Start first federation node**

For the 'bootstrapper' node you'll have to specify base admin credentials and network-credentials
- network credentials allow 'read-only' network operations and are required to be known for joining a network
- admin credentials are individual to each node, they are required to perform any elevated privilege action, e.g.: creating proxy routes

All can simply be specified like this:

```bash
./open-chat --default-network-credentials "<network-name>:<password>" --root-credentials "<user>:<password>"
```

You can also initalize not credentials without revealing them by specifying `--root-credentials "<username>:hashed_<pdf-hashed-password>"`.
Now that our bootstrapper node is available we can use its peer-id to bootstrap other nodes.

> If you want to persis your open-chat configuration system wide just run it with `--install`

We can extract it's peer Id by either loggin in to the UI with the admin user, or via the Open-Chat cli:

```bash
./open-chat client login                # you'll be promted to ender username and password
./open-chat client identity -b64        # revals the node id in a re-usable base64 format
eyJpZCI6ICI8TE9DQUxfUEVFUl9JRD4iLCAiY29ubmVjdF9tdWx0aWFkcmVzcyI6IFsiL2lwNC88U09NRV9JUF9BRFJFU1M+L3RjcC8wL3AycC88TE9DQUxfUEVFUl9JRD4iXX0=
```

#### **3. Adding other nodes to the network**

The only thing you need is the bootstrap peer string and the networks credentials

```bash
./open-chat -dnc "<network-name>:<password>" -rc "<different-node-specific-admin-creds>" -bs "<base64-bootstrap-peer-string>"
```

Now we just need to wait a moment untill these nodes found each other and established connections.
Lets see if the nodes are connected.

```bash
./open-chat client login          # ....
./open-chat client nodes -ls    # list all available nodes
```

Now lets *see if we can send requests to the node*:

```bash
./open-chat client get-metrics --node <remote-peer-id>
```

Hurray it works! **Now where do we go from here?**

#### **4. Establish on-demand proxy routing rules**

When we have a network, we can model its traffic in any-way we like.
Open-chat nodes can configure arbitrary routing rules on per-node basis, 
every node can only route traffic that goes trough itself.

To route traffic trough another node, routing rules on that node need to be established, 
thus the admin credentials for that node must be known.

Since we control the full network we know admin credentials of very node and can establish any routing rules we want.

**Lets use open-chat to create a ngrok-like TLS to behind NAT reverse-proxy**, to achieve this we need:
1. A single public peer in our control with an `<PUBLIC-IP>`
2. A domain name A-Record pointing that IP `<HOSTNAME>`
3. A private behind NAT node that hosts some TCP service on `<peer-port>`

> All the steps presented here can also be run from a single node, by using network-requests that are build in to Open-Chat.
> For simplicity we run command on individual nodes, but there are also workflows that allow controlling routing rule requests to ANY node just from a single network member.

##### 1. Use open-chat to request TLS certificate on the public node

```bash
./open-chat client tls --hostname "<HOSTNAME>" --key-prefix "<self-defined-certificate-prefix>"
```
This will automatically start the ACME challenge process and saves the obtained certificate in the Open-Chat nodes database.

##### 2. Establish TLS to network routing rules

First we need to tell the public node to accept TLS traffic using the just obtained certificate.
It should only accept traffic coming from `"<HOSTNAME>"` and the traffic should be routed to a specific port on the system.

```bash
./open-chat client proxy --direction "egress" --traffic-origin "<HOSTNAME>" --traffic-target "<key-prefix>:<peer-port>" --kind "tls" --key-prefix "<self-defined-certificate-prefix>"
```

This create an 'egress' proxy that accepts incoming traffic from the `"<HOSTNAME>"` and runnels it into the network.

```bash
./open-chat client proxy --direction "egress" --traffic-origin "<own-peer-id>:<peer-port>" --traffic-target "<remote-peer-id>:<peer-port>" --kind tcp
```

This establishes an 'ingress' rule from the network to the remote peer node.
Now the only thing that is left to do is the local node to tunnel network traffic from that peer to the `<peer-port>` that runs the TCP service.

##### 3. Accept Network Traffic on a local Port

Traffic is now routes from the domain to the public peer, from there inside the network and to the remote peer.
But currently the remote peer would just drop the traffic as it doesn't know what to do with it, lets route it to the port that runs our service:

```bash
./open-chat client proxy --direction "ingress" --traffic-origin "<own-peer-id>:<peer-port>" --traffic-target "<remote-peer-id>:<peer-port>" --kind tcp
```


### Conclusion and Outlook

**This Article merely gives a hint what Open-Chat Federation is able to do**, options from there go way further.
Just as food for though, the following thins are only a few lines of open-chat rules away:

1. **Decentralized Multi-Site behind NAT hosting**
Host a swarm of Server behind NAT on different locations, route them trough multiple public peers in a 'load balancing' fashion
and you have a load resilient decentralized server infrastructure that masks the actual server locations, allows for easy traffic filtering and allows for easy horizontal scaling by just extending the node pool.
2. **Persistent self hosted Ngrok**. As shown in this article open-chat can easily be used to create development routing rules as tools like ngrok can do.
3. **Self managed overlay networks for anything**. This connect extends to any application, also the ones fully privately easily use open-chat routing rules to manage traffic in your local network, *you may even use it in offline local networks!*.


How Open-Chats Federation Enables anybody to host anything anywhere

Networking

Expose NAT Services Across Two Kubernetes Clusters with Tailscale

Breaking Trough NAT: Decentralized Self-Hosting Via Open-Chat