# K3s Migration Plan — Project Level Up Javid

**Date:** 2026-08-17
**Author:** Hermes Agent (for Javid)
**Status:** ACTIVE — Path A+ is the canonical approach
**Objective:** Migrate the existing 19-service Docker Compose stack to a 2-node k3s cluster. First step toward CKA readiness + IDP-as-portfolio positioning.

---

## 🎯 The chosen approach: Path A+ — Tailscale-private 2-node cluster

**Philosophy:** Orchestrate everything from THIS box. The new node is a private compute extension reachable only via Tailscale. Zero public ports on the new node. SSH from this box becomes the k3s extension cord.

### Why this wins over a public Contabo VPS

| | Public Contabo VPS | **Path A+ Tailscale-private** |
|---|---|---|
| **Public exposure** | New node has public IP, needs firewall rules | **Zero public ports** — Tailscale only |
| **Monthly cost** | $5-8/mo | **$0** (old laptop/Pi) or $80 one-time (Pi 5 8Gi) |
| **Orchestration** | Manual SSH + remote kubectl | **Orchestrate from this box** — kubectl talks to remote API over Tailscale |
| **CKA alignment** | Multi-node ✓ | Multi-node ✓ — **PLUS** teaches private-cluster patterns |
| **Reversibility** | Kill VPS | Same — kill the 2nd node |
| **Security model** | VPS provider's network policies | **Tailnet ACLs** — only-this-box-can-SSH-to-control-plane |

### Hardware options (pick one)

| Option | Cost | Notes |
|---|---|---|
| **Old laptop** (4Gi+ RAM) | $0 | Repurpose an unused machine. Best for CKA prep. |
| **Raspberry Pi 5 (8Gi)** | $80 one-time | Realistic hardware, low power, fun for learning |
| **Friend's spare server** | $0 | Just needs Tailscale access + LAN |
| **Contabo VPS** (4Gi) | $5-8/mo | Only if you want a real datacenter node |

**My recommendation:** Old laptop or Pi 5. CKA exam is 100% software — it doesn't care about cloud vs. bare metal.

---

## 🏗️ Architecture

```
┌──────────────────────────────────────────────────────────────────┐
│                TAILNET (100.64.0.0/10, encrypted mesh)            │
│                                                                   │
│    ┌────────────────────────────┐  ┌──────────────────────────┐  │
│    │  THIS BOX                  │  │  NEW NODE                │  │
│    │  (worker-1)                │  │  (control-plane-1)       │  │
│    │                            │  │                          │  │
│    │  • 19 Docker services      │  │  • k3s control-plane    │  │
│    │  • k3s agent (worker)      │◄─┤  • etcd (single node)   │  │
│    │  • kubectl → remote API    │  │  • No public ingress    │  │
│    │  • Mission control hub     │  │  • No public IP in DNS  │  │
│    │                            │  │  • Tailscale-only access │  │
│    └────────────────────────────┘  └──────────────────────────┘  │
│              ▲                                ▲                  │
│              │ Tailscale ACL: tag:k3s-worker  │                  │
│              │ can SSH to tag:k3s-cp         │                  │
└──────────────────────────────────────────────────────────────────┘
```

---

## 📅 5-week timeline

| Week | Phase | What happens |
|---|---|---|
| **1** | Pre-migration | 2nd node provisioned, Tailscale joined, k3s installed on both, ACL locked down |
| **2** | Admin UIs | Migrate 5 stateless admin tools (homarr, portainer, pocket-id, glances, redisinsight) |
| **3** | Stateful | Migrate 4 stateful services (redis, gitea, postgres, authentik-redis) |
| **4** | Critical | Migrate 4 critical services (authentik, open_notebook, n8n, vectordb) |
| **5** | Long-tail | Cleanup, crowdsec stays on host |
| **6+** | CKA study | Use the live cluster as your CKA exam lab |

---

## 📋 Step-by-step install

### Step 1: Provision the 2nd node

Pick your hardware. For this plan I'll assume Ubuntu 24.04 (any Debian-family works).

```bash
# === On the new node, via physical keyboard or initial SSH ===
# (First-time SSH will need the new node's public IP temporarily,
#  then we lock everything down via Tailscale after install)

# Install base packages
sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget git vim

# Set hostname
sudo hostnamectl set-hostname cp-1
```

### Step 2: Install Tailscale on the new node + tag it

```bash
# On the new node
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --authkey=tskey-auth-XXXXX \
                   --hostname=cp-1 \
                   --advertise-tags=tag:k3s-cp

# Verify it joined
tailscale status
# Should show both this box (worker-1) and cp-1 as connected
```

### Step 3: Install Tailscale on this box with the worker tag

```bash
# On THIS box (you may already be tagged — check first)
sudo tailscale status | grep -E "tag:|100\."

# If you need to add the tag (one-time):
sudo tailscale up --advertise-tags=tag:k3s-worker

# Verify MagicDNS works
ping -c 3 cp-1
# Should resolve to 100.X.Y.Z
```

### Step 4: Set up the Tailscale ACL (lock it down)

See `k3s-acl.json` in this directory. Apply it via the Tailscale admin console:
https://login.tailscale.com/admin/acls/file

The ACL enforces:
- `tag:k3s-worker` (this box) can SSH to `tag:k3s-cp` (control-plane)
- `tag:k3s-worker` can hit the k3s API on `tag:k3s-cp:6443`
- `tag:k3s-cp` can reach kubelet on `tag:k3s-worker:10250`
- Everything else is denied by default

### Step 5: Install k3s on the new node (control-plane)

```bash
# === From THIS box, SSH into the new node ===
ssh cp-1

# On the new node, disable swap (k3s requires it)
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

# Install k3s server with Tailscale-friendly settings
curl -sfL https://get.k3s.io | \
  INSTALL_K3S_EXEC="--tls-san $(tailscale ip -4)
                    --disable traefik
                    --disable servicelb
                    --disable local-storage
                    --node-name cp-1
                    --flannel-backend=none
                    --cluster-cidr=10.42.0.0/16
                    --service-cidr=10.43.0.0/16" \
  sh -

# Verify k3s is running
sudo systemctl status k3s
sudo kubectl get nodes   # should show cp-1 as Ready

# Save the join token (we'll use this from this box)
sudo cat /var/lib/rancher/k3s/server/node-token
# K10XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX::server:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

# Exit back to this box
exit
```

### Step 6: Install k3s agent on THIS box (worker)

```bash
# === On THIS box ===
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

# Save the join token to a local file (paste from Step 5)
echo "K10XXX...::server:XXX..." > ~/.k3s-join-token

# Install k3s agent in worker mode
curl -sfL https://get.k3s.io | \
  K3S_URL=https://$(tailscale ip -4 cp-1):6443 \
  K3S_TOKEN=$(cat ~/.k3s-join-token) \
  INSTALL_K3S_EXEC="--node-name worker-1" \
  sh -

# Verify from this box
sudo kubectl get nodes
# NAME      STATUS   ROLES                  AGE   VERSION
# cp-1      Ready    control-plane,master   5m    v1.30.4+k3s1
# worker-1  Ready    <none>                 1m    v1.30.4+k3s1
```

### Step 7: Set up kubeconfig locally (orchestration from this box)

```bash
# From this box, copy the kubeconfig from the new node
mkdir -p ~/.kube
scp cp-1:/etc/rancher/k3s/k3s.yaml ~/.kube/config-cp-1

# Edit the server line to use the Tailscale IP (not localhost)
sed -i "s|server: https://127.0.0.1:6443|server: https://$(tailscale ip -4 cp-1):6443|" ~/.kube/config-cp-1

# Set as default kubeconfig (or alias it)
echo 'export KUBECONFIG=~/.kube/config-cp-1' >> ~/.bashrc
source ~/.bashrc

# Test from this box
kubectl get nodes -o wide
# Both nodes should show up with their Tailscale IPs
```

### Step 8: Install Tailscale CNI (replaces Flannel)

This is optional but recommended — keeps zero-public-ports invariant even for pod-to-pod traffic.

```bash
# On this box (since we're orchestrating from here)
helm repo add tailscale https://tailscale.github.io/tailscale-operator
helm repo update

helm install tailscale-operator tailscale/tailscale-operator \
  --namespace tailscale \
  --create-namespace \
  --set oauth.clientId=ts-client-id-XXXXX \
  --set oauth.secret=ts-secret-XXXXX
```

### Step 9: Install Longhorn for replicated storage

```bash
# On this box
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml

# Wait for it to come up
kubectl -n longhorn-system get pods
# All should be Running
```

---

## 🚚 Migration phases (Weeks 2-5)

### Phase 1 — Low-risk admin UIs (Week 2)

These are non-critical, restartable, no state. Migrate first to validate the pipeline.

**Services to migrate:**
1. `app-homarr` (410Mi) → `homarr` namespace
2. `app-portainer` (96Mi) → `admin` namespace
3. `app-pocket-id` (87Mi) → `auth` namespace
4. `app-glances` (100Mi) → `monitoring` namespace
5. `app-redisinsight` (108Mi) → `tools` namespace

**Rollback strategy:**
- Keep Docker Compose running on this box until k3s version is verified
- Use `kubectl get svc -o wide` to confirm ClusterIP works
- For external access, use Tailscale ingress annotation
- If something breaks: `docker compose up -d` brings service back in 5 seconds

**Conversion pattern (example for Homarr):**

```yaml
# homarr-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: homarr
  namespace: admin
spec:
  replicas: 1
  selector:
    matchLabels: { app: homarr }
  template:
    metadata:
      labels: { app: homarr }
    spec:
      containers:
      - name: homarr
        image: ghcr.io/homarr-labs/homarr:v1.74.0
        ports: [{ containerPort: 7575 }]
        resources:
          requests: { memory: "256Mi", cpu: "100m" }
          limits: { memory: "512Mi", cpu: "500m" }
---
apiVersion: v1
kind: Service
metadata:
  name: homarr
  namespace: admin
  annotations:
    tailscale.com/expose: "true"   # Tailscale ingress
spec:
  selector: { app: homarr }
  ports: [{ port: 80, targetPort: 7575 }]
```

### Phase 2 — Stateful services (Week 3)

**Services to migrate:**
6. `redis` (small, ~50Mi) → `cache` namespace
7. `app-gitea` (111Mi + postgres) → `git` namespace
8. `db-authentik` (postgres) → `auth` namespace
9. `redis-authentik` (small) → `auth` namespace

**State persistence:** Use **Longhorn** (k3s-native CSI driver) for replicated block storage across the 2 nodes.

**Rollback strategy:**
- **Snapshot BEFORE migration**: Longhorn snapshot of any PVC
- **Restore AFTER migration**: `kubectl apply -f snapshot-restore.yaml`
- **Database migration**: `pg_dump` → restore via k8s Job, verify checksum
- **Keep docker volumes untouched** until k8s version validated for 48h

### Phase 3 — Critical services (Week 4)

**Services to migrate:**
10. `app-authentik` + `worker-authentik` → `auth` namespace
11. `openotebk-open_notebook-1` + `openotebk-surrealdb-1` → `notebook` namespace
12. `n8n` (400Mi) → `automation` namespace
13. `vectordb-qdrant` + `vectordb-embeddings` → `vectors` namespace

**This phase requires downtime window:** Plan for 30-60 min per service.

### Phase 4 — Long-tail cleanup (Week 5)

- `app-audiobookshelf` (rarely used)
- `app-ignis` (file sharing)
- `agent-vault` (custom, low priority)
- `crowdsec` + `crowdsec-firewall-bouncer` — **KEEP ON HOST** (kernel-level, not containerizable)

---

## 🛡️ Rollback strategy (whole-cluster)

If the whole k3s experiment goes sideways, you have two rollback options:

### Option R1: Soft rollback (5 minutes, zero data loss)

```bash
# On the new node (via SSH)
ssh cp-1 "sudo systemctl stop k3s"

# On this box
sudo systemctl stop k3s-agent

# All Docker Compose services are STILL RUNNING — they were never stopped
docker compose ps   # should show all 19 services healthy
```

**Risk:** Zero. Docker Compose was running in parallel the whole time.

### Option R2: Full teardown (15 minutes)

```bash
# On this box (worker)
sudo /usr/local/bin/k3s-uninstall.sh

# On new node (control-plane)
ssh cp-1 "sudo /usr/local/bin/k3s-uninstall.sh"

# Remove Longhorn, Tailscale CNI (if installed via Helm)
helm uninstall longhorn -n longhorn-system
helm uninstall tailscale-operator -n tailscale
```

---

## ✅ Risk mitigation checklist

Before starting:
- [ ] **Backups verified restorable**: Restore docker volume backup to a separate test path
- [ ] **DNS rollback plan**: If MagicDNS resolution fails, have static /etc/hosts entries ready
- [ ] **Authentik export**: Run `authentik export` to dump IdP config (critical for SSO continuity)
- [ ] **Gitea backup**: `gitea dump` to single tarball (mirror this to off-host first)
- [ ] **Postgres dumps**: `pg_dump` for each DB, checksum-verified
- [ ] **Tailscale ACL applied**: See `k3s-acl.json` in this directory

---

## 🎓 CKA exam alignment

This migration plan covers the following CKA exam domains (CNCF 2026 syllabus):
- **Cluster Architecture (15%)**: Setup + HA — ✅ Multi-node setup with k3s
- **Workloads & Scheduling (20%)**: Deployments, StatefulSets — ✅ Phase 2-3
- **Services & Networking (20%)**: Ingress, NetworkPolicy — ✅ Tailscale CNI
- **Storage (10%)**: PVs, PVCs, StorageClasses — ✅ Longhorn
- **Troubleshooting (30%)**: kubectl logs, debug — ✅ Built into all phases

After this migration, you'll have hands-on experience with ~80% of the CKA exam material.

---

## 💸 Cost

- **Hardware**: $0 (old laptop) or $80 one-time (Pi 5 8Gi)
- **Time**: ~30-40 hours over 5-6 weeks (5-8h/week)
- **Risk**: LOW — Docker Compose keeps running in parallel throughout

---

## 🚀 Post-CKA

Once CKA passes:
- Keep the 2-node cluster running (cost is minimal)
- Use it as a permanent IDP/Backstage dev environment
- Document the journey as a **portfolio piece** (blog post + case study)
- IDP/Backstage layer added next (per the Q3 + Q4 IDP gap signal)
- Consider adding a 3rd node for true HA (control-plane + 2 workers)

---

## 📂 Related files in this directory

- `k3s-acl.json` — Tailscale ACL policy for the cluster
- `case-study.md` — Public writeup of the migration journey