# Tailscale + Kubernetes: Architecture Comparison

**Date:** 2026-08-21
**Author:** Hermes Agent (for Javid)
**Context:** Self-hosted 19-service stack migration to a 2-node Tailscale-private k3s cluster
**Question:** "Can Kubernetes (k3s) apps be deployed behind a Tailscale network? Does Tailscale work at all with k3s?"

**Short answer:** Yes — absolutely. The canonical production pattern for "private apps on the tailnet, zero public ports" is **Pattern C: a reverse proxy (Traefik/Caddy) joined to Tailscale, with standard K8s Ingress for routing** — NOT Tailscale-as-CNI.

---

## 🎯 The mental model (recommended pattern)

```
                          Internet
                              │
                              │ (zero public ports — nothing reachable)
                              │
                              │
                         ┌────▼────┐
                         │   K3s   │
                         │ Cluster │
                         │         │
                         │ ┌─────┐ │
                         │ │ TS  │ │ ← Single Tailscale sidecar
                         │ │ Side│ │   (Traefik pod)
                         │ └──┬──┘ │   Hostname: traefik-ingress
                         │    │    │
                         │ ┌──▼──┐ │
                         │ │Homarr│ │ ← Pod with private K8s IP (10.42.x.x)
                         │ └─────┘ │
                         │ ┌─────┐ │
                         │ │Gitea│ │ ← Pod with private K8s IP (10.42.x.x)
                         │ └─────┘ │
                         └────┬────┘
                              │
                              │ WireGuard-encrypted tunnel
                              │
                         ┌────▼────┐
                         │ Tailnet │
                         └────┬────┘
                              │
              ┌───────────────┼───────────────┐
              │               │               │
          �───▼───┐       ┌───▼───┐       ┌───▼───┐
          │Laptop │       │ Phone │       │Other  │
          │       │       │       │       │servers│
          └───────┘       └───────┘       └───────┘
          (routes via Traefik on tailnet — homarr.ts.net, gitea.ts.net)
```

**The key insight:** Tailscale doesn't need to be the cluster's network. A single reverse proxy on the tailnet handles all service routing, leaving K8s networking to do what it does best.

---

## 🏆 The five patterns (ranked)

### 🥇 Pattern C — Reverse Proxy + Tailscale (RECOMMENDED)

**What it is:** Standard k3s cluster with Flannel/Cilium CNI. A single reverse proxy (Traefik or Caddy) runs as a Tailscale sidecar. All services are exposed via standard K8s Ingress routed through the reverse proxy.

**How it works:**
1. Standard k3s install with default Flannel CNI
2. Install Traefik (k3s bundles it) as the ingress controller
3. Add a Tailscale sidecar to the Traefik deployment
4. Configure Traefik ingresses per service with TLS via Tailscale certs
5. Each service is reachable on the tailnet as `service.ts.net`

**Architecture:**
```
┌──────────────────────────────────────────────────────────────�
│  K3s Cluster (this box + new VPS)                            │
│                                                               │
│  �─────────────────────────────────────────────────────┐    │
│  │ Deployment: traefik (with Tailscale sidecar)        │    │
│  │   Traefik pod:                                        │    │
│  │     IP: 10.42.0.10 (Flannel) + 100.64.X.Y (TS)     │    │
│  │   Sidecar pod:                                        │    │
│  │     Hostname: traefik-ingress                       │    │
│  │     IP: 100.64.X.Y (single Tailscale node)         │    │
│  │   Routes (Ingress resources):                         │    │
│  │     homarr.ts.net  → homarr.default.svc.cluster.local│    │
│  │     gitea.ts.net   → gitea.default.svc.cluster.local │    │
│  │     grafana.ts.net → grafana.monitoring.svc.cluster  │    │
│  └─────────────────────────────────────────────────────┘    │
│                                                               │
│  ┌─────────────────────────────────────────────────────┐    │
│  │ Service: homarr (no Tailscale needed)               │    │
│  │   Pod: 10.42.0.5                                     │    │
│  │   Service: homarr.default.svc.cluster.local         │    │
│  │   Ingress: homarr.ts.net → traefik-ingress          │    │
│  └─────────────────────────────────────────────────────┘    │
│                                                               │
│  ┌─────────────────────────────────────────────────────�    │
│  │ Service: gitea (no Tailscale needed)                │    │
│  │   Pod: 10.42.0.6                                     │    │
│  │   Service: gitea.default.svc.cluster.local          │    │
│  └─────────────────────────────────────────────────────�    │
└──────────────────────────────────────────────────────────────┘
                              │
                              ▼
                    ┌──────────────────┐
                    │     Tailnet      │
                    │   100.64.0.0/10  │
                    └──────────────────┘
```

**Pros:**
- ✅ **1 Tailscale node total** (just Traefik) — negligible license usage
- ✅ **Standard K8s networking** — Flannel/Cilium do their thing, no CNI changes
- ✅ **Standard K8s Ingress** — Traefik Ingress resources, well-documented
- ✅ **Mature ecosystem** — Traefik dashboard, Let's Encrypt integration, Tailscale certs
- ✅ **Per-service ACL** — Traefik middleware + Tailscale ACL tags
- ✅ **Production-ready** — pattern used by most companies with private k8s
- ✅ **Easy HA** — 2 Traefik replicas behind a LoadBalancer or DNS round-robin
- ✅ **CKA-aligned** — Ingress is CKA syllabus domain (15% weight)
- ✅ **Migration-friendly** — same pattern as your existing Traefik-on-Docker setup

**Cons:**
- ❌ Single ingress point (mitigated with replicas + health checks)
- ❌ All services share the same Tailscale IP (Traefik's)
- ❌ TLS termination at Traefik (not at each pod)
- ❌ Need to manage Ingress resources (Traefik CRDs)

**Best for:** Your case. Self-hosted stack, zero-public-ports, modern private-cluster pattern, CKA prep, Senior PE portfolio.

**Install commands (k3s has Traefik built-in):**
```bash
# k3s auto-installs Traefik. Add a Tailscale sidecar via Helm values:
cat > /var/lib/rancher/k3s/server/manifests/traefik.yaml <<EOF
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: traefik
  namespace: kube-system
spec:
  chart:
    spec:
      chart: traefik
      sourceRef:
        kind: HelmChart
        name: traefik
        namespace: kube-system
  values:
    additionalArguments:
      - --entrypoints.websecure.http.tls.certResolver=letsencrypt
    ingressRoute:
      dashboard: enabled: true
    ports:
      websecure:
        tls:
          enabled: true
    providers:
      kubernetesIngress: enabled: true
    # Tailscale sidecar config (added as a separate pod)
EOF

# Deploy Tailscale sidecar separately
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik-tailscale
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik-ts
  template:
    metadata:
      labels:
        app: traefik-ts
    spec:
      containers:
      - name: traefik-tailscale
        image: tailscale/tailscale:latest
        env:
        - name: TS_AUTHKEY
          value: tskey-auth-XXXXX
        - name: TS_HOSTNAME
          value: traefik-ingress
        - name: TS_AUTH_ONCE
          value: "true"
        - name: TS_USERSPACE
          value: "false"
        securityContext:
          capabilities:
            add: [NET_ADMIN]
      restartPolicy: Always
EOF

# Add an Ingress per service
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: homarr
  namespace: default
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
spec:
  rules:
  - host: homarr.ts.net
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: homarr
            port:
              number: 7575
  tls:
  - hosts:
    - homarr.ts.net
    secretName: homarr-tls
EOF
```

---

### 🥈 Pattern B — Tailscale Sidecar (EASIEST)

**What it is:** Each pod has a Tailscale sidecar (same pattern as your existing Docker Compose stack). Standard K8s networking, but every pod has a sidecar on the tailnet.

**How it works:**
- Each Deployment has 2 containers: app + `tailscale/tailscale:latest`
- The Tailscale sidecar joins the tailnet with `--auth-once=true`
- App reaches the tailnet via `localhost:port` (sidecar's userspace networking)
- Each service gets its own Tailscale hostname

**Architecture:**
```
�──────────────────────────────────────────────────────────────┐
│  K3s Cluster (this box + new VPS)                            │
│                                                               │
│  ┌─────────────────────────────────────────────────────┐    │
│  │ Pod: homarr-7d4f8b9c-x7kz2                         │    │
│  │                                                      │    │
│  │  ┌─────────────────────┐  ┌──────────────────────┐  │    │
│  │  │ Container: homarr   │  │ Container:           │  │    │
│  │  │ Image: ghcr.io/...  │  │ Image: tailscale/    │  │    │
│  │  │ Port: 7575          │◄─┤ tailscale:latest     │  │    │
│  │  │                     │  │ Hostname: ts-homarr  │  │    │
│  │  └─────────────────────┘  └──────────────────────┘  │    │
│  │   (Pod IP: 10.42.0.5, Flannel)                       │    │
│  └─────────────────────────────────────────────────────┘    │
│                                                               │
│  ┌─────────────────────────────────────────────────────┐    │
│  │ Pod: gitea-restart-0                                │    │
│  │  �─────────────────────┐  ┌──────────────────────┐  │    │
│  │  │ Container: gitea    │  │ Container:           │  │    │
│  │  │ Image: gitea/gitea  │◄─┤ tailscale:latest     │  │    │
│  │  └─────────────────────┘  └──────────────────────┘  │    │
│  └─────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────┘
                              │
                              ▼
                    ┌──────────────────┐
                    │     Tailnet      │
                    │  (one node/pod)  │
                    └──────────────────┘
```

**Pros:**
- ✅ **Same pattern as your existing Docker Compose stack** — zero learning curve
- ✅ **Sidecar survives pod restarts** (via `--auth-once=true`)
- ✅ **Works with any CNI** (Flannel, Calico, Cilium, etc.)
- ✅ **Easy to reason about** — each pod is self-contained
- ✅ **Minimal install** — just one Deployment template + auth key
- ✅ **No Traefik/Ingress needed** — apps are directly on the tailnet

**Cons:**
- ❌ **N Tailscale nodes** for N pods (eats 100-device cap faster)
- ❌ **2x container resources per app** (app + sidecar)
- ❌ **Sidecar auth key management** — re-usable key, not one-shot
- ❌ **Sidecar scale issues** — 50 pods = 50 Tailscale nodes

**Best for:** Quick migration. Same pattern as your Docker Compose stack. Fast wins.

**Pod template example:**
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: homarr
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: homarr
        image: ghcr.io/homarr-labs/homarr:v1.74.0
        ports: [{ containerPort: 7575 }]
      - name: tailscale
        image: tailscale/tailscale:latest
        env:
        - name: TS_AUTHKEY
          value: tskey-auth-XXXXX
        - name: TS_HOSTNAME
          value: ts-homarr
        - name: TS_AUTH_ONCE
          value: "true"
        - name: TS_USERSPACE
          value: "false"
        securityContext:
          capabilities:
            add: [NET_ADMIN]
```

---

### 🥉 Pattern A — Tailscale CNI (BEST FOR SOME, NOT YOURS)

**What it is:** Replace k3s's default Flannel CNI with the Tailscale CNI plugin. Every pod gets a Tailscale IP.

**How it works:**
- Install the Tailscale Operator as a Deployment
- Apply annotation to Services: `tailscale.com/expose: "true"`
- The Operator creates a Tailscale node per Service
- ClusterIP service becomes reachable on the tailnet

**Architecture:**
```
┌──────────────────────────────────────────────────────────────┐
│  K3s Cluster (this box + new VPS)                            │
│                                                               │
│  ┌─────────────────────────────────────────────────────┐    │
│  │ Pod: homarr-7d4f8b9c-x7kz2                         │    │
│  │   IP: 100.64.5.42 (Tailscale)                       │    │
│  │   Service: homarr.default.svc.cluster.ts.net        │    │
│  │   Annotations:                                        │    │
│  │     tailscale.com/expose: "true"                    │    │
│  └─────────────────────────────────────────────────────�    │
│                                                               │
│  ┌─────────────────────────────────────────────────────┐    │
│  │ Pod: gitea-restart-0                                │    │
│  │   IP: 100.64.5.43 (Tailscale)                       │    │
│  │   Service: gitea.default.svc.cluster.ts.net         │    │
│  └─────────────────────────────────────────────────────┘    │
│                                                               │
│  ┌─────────────────────────────────────────────────────┐    │
│  │ Deployment: tailscale-operator                      │    │
│  └─────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────┘
```

**Pros:**
- ✅ **Every pod is a first-class Tailscale device** with its own IP + DNS name
- ✅ **Service discovery via MagicDNS**
- ✅ **Per-pod ACL** — most granular control

**Cons:**
- ❌ **N Tailscale nodes** for N pods (license pressure)
- ❌ **WireGuard encryption** per pod = CPU overhead
- ❌ **Init containers may fail DNS** — tricky edge cases
- ❌ **Not the canonical production pattern** — most clusters use a reverse proxy
- ❌ **Harder to debug** — pod-level Tailscale state is harder to inspect
- ❌ **CNI swap risk** — replacing Flannel is invasive
- ❌ **Replacing Traefik with Tailscale** means losing Traefik's mature ecosystem (Let's Encrypt, dashboard, middleware)

**Best for:** Clusters where you NEED per-pod ACLs (rare). Multi-tenant clusters with strict isolation requirements. Not your case.

---

### Pattern D — Tailscale Funnel (PUBLIC)

**What it is:** Tailscale-managed public ingress via Funnel.

**Pros:** Zero ops
**Cons:** Defeats the zero-public-ports goal — services become publicly accessible via `*.ts.net`

**Best for:** Public-facing apps where you WANT them on the internet.

**Verdict:** ❌ Skip for your case.

---

### Pattern E — Headscale + custom mesh

**What it is:** Self-hosted Tailscale control server.

**Pros:** No Tailscale license, full control
**Cons:** More ops, less mature

**Best for:** Companies with strict data residency requirements or who don't want vendor lock-in.

**Verdict:** ❌ Overkill for your case. Stick with Tailscale.

---

## 📊 Side-by-side comparison

| Decision factor | **C: Reverse Proxy** ⭐ | **B: Sidecar** | **A: Tailscale CNI** |
|---|---|---|---|
| **Per-pod Tailscale IP** | ❌ No | ❌ No (sidecar only) | ✅ Yes |
| **Per-Service ACL granularity** | ✅ Via Traefik middleware | ⚠️ Sidecar-tag only | ✅ Per-pod tags |
| **Tailscale nodes consumed** | 🟢 **1** | 🔴 N (= N pods) | 🔴 N (= N pods) |
| **CPU overhead per pod** | 🟢 None (only at proxy) | 🟡 Medium (2 containers) | 🟡 Low (WireGuard in kernel) |
| **Memory overhead per pod** | � None | 🟡 Medium (~50MB sidecar) | 🟢 Low (~10MB) |
| **License usage at scale** | 🟢 Negligible | � Eats 100 cap | 🔴 Eats 100 cap |
| **Install complexity** | 🟢 Low (built-in Traefik) | 🟢 Low (sidecar template) | 🟡 Medium (Helm + operator) |
| **Failure domain** | 🟢 HA via 2 replicas | 🟢 Per-pod | 🟢 Per-pod |
| **TLS termination** | � Traefik (Let's Encrypt + TS certs) | 🟡 Per-sidecar | � Per-sidecar |
| **Observability** | 🟢 Traefik dashboard | � Per-sidecar | 🟡 Tailscale admin |
| **CKA alignment** | ✅ **Best** (Ingress = 15%) | 🟡 Sidecar is legacy | 🟡 CNI is CKA domain |
| **Production readiness** | ✅ Industry standard | ⚠️ Self-managed | ⚠️ Niche |
| **Migration story** | 🟢 Standard K8s patterns | 🟢 Lift-and-shift | 🟡 CNI swap |
| **Debug difficulty** | � Easy (standard K8s) | 🟢 Easy | 🔴 Hard (CNI swap) |
| **Multi-cluster** | 🟢 Works | 🟢 Works | 🟡 Federation complex |
| **HA story** | 🟢 2 Traefik replicas | 🟢 Per-pod restart | 🟢 Per-pod restart |
| **Service mesh compatible** | ✅ Yes | ✅ Yes | ⚠️ Conflicts |

---

## 🎯 My recommendation for Project Level Up Javid

**Goal:** Migrate 19 services from Docker Compose to k3s, zero public ports, 2-node cluster, CKA prep, Senior PE portfolio.

### Recommended path: B (Week 2-3) → C (Week 4-5)

### Why hybrid?

1. **Phase 1 (Week 2-3)** — Quick migration with Pattern B (sidecar):
   - Same pattern as your existing Docker Compose stack
   - Migrate 19 services in 1-2 weeks
   - Each pod has a Tailscale sidecar = accessible on tailnet
   - Minimal install upfront

2. **Phase 2 (Week 4-5)** — Transition to Pattern C (Reverse Proxy + Tailscale):
   - Install Traefik (already bundled with k3s)
   - Add Tailscale sidecar ONLY to Traefik (1 node total)
   - Migrate each service to use Ingress resources
   - Standard K8s pattern = CKA-aligned
   - Demonstrates production-ready architecture for portfolio

3. **Phase 3 (Post-CKA)** — Optionally explore Pattern A (Tailscale CNI):
   - Only if you need per-pod ACLs (you don't, currently)
   - Good for showing CNI mastery on the CKA exam

### Why Pattern C is the right long-term answer

- ✅ **Most production k3s clusters use this pattern** — it's what hiring managers expect
- ✅ **1 Tailscale node** vs 19+ — license-friendly
- ✅ **Standard K8s** — easier to hire help, easier to debug
- ✅ **Traefik is the de facto k3s ingress** — built-in, no extra install
- ✅ **CKA-aligned** — Ingress is on the syllabus
- ✅ **Portfolio-friendly** — "Migrated to k3s with Traefik + Tailscale ingress" is a senior-tier story

### Why NOT Pattern A (Tailscale CNI) for your case

- ❌ Per-pod ACLs is overkill — you have 1 user (you)
- ❌ Replaces Traefik with Tailscale — loses mature ecosystem
- ❌ Harder to debug, less production-typical
- ❌ 19 Tailscale nodes vs 1 — license pressure
- ❌ CNI swap is invasive — risky migration

---

## ⚠️ What WON'T work

| Pattern | Why it breaks |
|---|---|
| **Tailscale CNI + Docker `network_mode: service:tailscale`** | Sidecar mixes netns. With CNI, you need per-pod IPs. Pick one. |
| **Tailscale sidecar without `--auth-once=true`** | Pod restart = orphaned Tailscale node. Always use auth-once. |
| **Tailscale sidecar without `NET_ADMIN` capability** | WireGuard won't init. Need to grant `capabilities.add: [NET_ADMIN]`. |
| **Traefik ingress without Tailscale cert resolver** | TLS will fail. Use Tailscale's built-in cert magic or Let's Encrypt. |
| **Tailscale Funnel for private services** | Defeats the zero-public-ports goal. |
| **Tailscale + multi-cluster without federation** | Each cluster is its own tailnet node. Federation requires extra config. |

---

## 🔗 Migration plan alignment

Your existing `~/.hermes/career-levelup-javid/plans/k3s-migration-plan.md` Step 8 references Tailscale. Updated plan:

**Step 8 (revised — Pattern C preferred):**
```bash
# Phase 1 (Week 2-3): Tailscale sidecar pattern (Pattern B)
# Each migrated service uses the same pattern as Docker Compose
# - No CNI changes needed
# - Each pod has tailscale/tailscale:latest sidecar
# - Reachable on tailnet via <service>-<namespace>.ts.net

# Phase 2 (Week 4-5): Migrate to Pattern C (Reverse Proxy + Tailscale)
# 1. Traefik is already installed by k3s
# 2. Add Tailscale sidecar to Traefik deployment
# 3. Migrate services from sidecar pattern to Ingress resources
# 4. Standard K8s — easier to debug, CKA-aligned
```

**ACL updates (add to `k3s-acl.json`):**
```json
{
  "action": "accept",
  "src": ["tag:k3s-worker"],
  "dst": ["tag:k3s-cp:22"]
},
{
  "action": "accept",
  "src": ["tag:k3s-worker"],
  "dst": ["tag:k3s-cp:6443"]
},
{
  "action": "accept",
  "src": ["autogroup:admin"],
  "dst": ["tag:k3s-ingress:443", "tag:k3s-ingress:80"]
}
```

Add a new tag: `tag:k3s-ingress` for the Traefik pod.

---

## 💡 Bottom line

**Yes, Tailscale + k3s absolutely works.** It's a Tier-1 integration.

**For your goal — "apps private, accessible from laptop/phone, zero public ports" — the canonical production pattern is Pattern C (Reverse Proxy + Tailscale).** It's:
- ✅ Industry standard
- ✅ 1 Tailscale node (vs 19+ with CNI/sidecar)
- ✅ CKA-aligned (Ingress is syllabus)
- ✅ Production-ready
- ✅ Easy to debug
- ✅ Senior PE portfolio piece

**Use Pattern B for the quick migration (Week 2-3), then transition to Pattern C (Week 4-5).** Pattern A is overkill for your use case.

---

## 📂 See also

- `k3s-migration-plan.md` — Main migration plan (Step 8: Pattern C setup)
- `k3s-acl.json` — Tailscale ACL policy (needs `tag:k3s-ingress` added)
- `case-study.md` — Public-facing case study
- [Traefik k8s docs](https://doc.traefik.io/traefik/providers/kubernetes-ingress/)
- [Tailscale cert magic](https://tailscale.com/kb/1153/enabling-https)
- [k3s built-in Traefik](https://docs.k3s.io/networking)

---

*This doc is part of the Project Level Up Javid portfolio. As you implement, update the case study with real benchmarks: install time, node count, latency, etc.*