A real migration from Docker Compose to k3s, where every workload is reachable only over an encrypted Tailscale mesh — zero public ports. Built as CKA exam prep and Senior Platform Engineer portfolio work.
dockhand (the docker-management UI, deliberately aborted).
The other compose directories are dormant dev configs with no running containers.
The headline milestone: a live production service moved with zero downtime, from a systemd unit to a 2-replica Deployment. HA from day one.
First stateful service. SQLite plus repositories migrated to a hostPath PVC, data integrity verified by row count before the old container was decommissioned.
A database and its consumer migrated as one unit, so neither was left
unable to reach the other. Credentials moved from a .env file to a Secret.
The headline architectural win. A whole Tailscale sidecar container — with kernel capabilities, its own tailnet identity, and its own auth key — replaced by one declarative field.
First multi-container pod: MLflow plus a Caddy basic-auth proxy as in-pod sidecars. The Service deliberately targets the proxy's port so auth cannot be bypassed.
Two Deployments talking over Service DNS. This is also the redemption run: the same service had failed outright on Compose after 18 iterations.
Every AI workload now routes through one in-cluster gateway instead of each holding provider keys. Repaired from 3 broken routes to 13 of 14 working.
Self-hosted Obsidian vault sync, tailnet-only — keeps native Obsidian on desktop and mobile while replacing only the paid sync service. Chosen over browser-based alternatives; one container, one port.
require_valid_user applies to every request including
/_up, so a k8s httpGet probe gets 401 — and the
liveness probe then kills a perfectly healthy database in a restart loop. Fixed with the
require_valid_user_except_for_up carve-out and authenticated
exec probes.
Not a migration — a local-first build. A Bitwarden-compatible server was shared as a Cloudflare Workers project; it was evaluated and skipped because a Worker is a public endpoint, and the native Bitwarden clients cannot present Cloudflare Access tokens, so it could not be made tailnet-only. Built the equivalent on k3s instead: same official clients, zero public ports, no vendor data plane.
Recreate rollout the
vault's rsa_key.pem stayed byte-identical and all 29 tables
remained on the PV — that is what proves the mount and persistence actually work.
curl at the websocket route returned 404, which
looked like a broken endpoint. With the upgrade headers and
--http1.1 it returns 401 — the route was fine all along.
Test the protocol the clients use, not a convenient approximation of it.
The original stack reached the tailnet by giving every service its own Tailscale sidecar container. Kubernetes collapses all of that into a single declarative field.
docker cp nesting trap
·
hit 3 separate times
RECURRING
docker cp svc:/data - preserves the data/ parent,
so data lands one level too deep. The pod boots with an empty volume and shows a
fresh-install screen instead of real data.
# wrong — nests data/data/ docker cp svc:/data - | tar xf - -C <pv> # correct — trailing /. on the SOURCE docker cp svc:/data/. - | tar xf - -C <pv> find <pv> -maxdepth 2 -type d # always verify before applying
A file-reading service was migrated with its data
COPY-ed into the image and no volume mount, so
main.py read the archive frozen at build time while the cron kept
writing fresh data to the host. Pods 1/1 Running, ingress
200, HTML rendering correctly — and the content three days old.
127.0.0.1:8890) rather than the Tailscale ingress people
actually browse, so it could only ever detect a crash — never staleness or
a wrong route. A second near-miss: the page did show a current date,
but that was the sidebar date picker, not the data. Comparing
newest-served against newest-on-disk is what exposed it.
/app so code and data both
come from the host (matching the systemd unit's
WorkingDirectory); rebuild the watchdog with two lanes —
user-facing liveness and data freshness, with an unreadable probe
reported as UNKNOWN rather than silently passing.
The first nesting repair copied a parent directory into its own child, then deleted the source — leaving an empty directory. The data looked gone.
find before touching anything.
reset is global, not per-mount. Used it to remove one path
and wiped every other serve on the box. All six were restored manually.
Two pods cannot share one host port. The scheduler's message implies the port is taken on the host — it is not; the scheduler simply refuses to bind two pods to it.
Two routes returned Invalid API Key for weeks. The config was
correct — the env var held the wrong provider's key:
GROQ_API_KEY contained an xai- key, not a gsk_ one.
XAI_API_KEY, and
left GROQ_API_KEY deliberately empty so availability checks report
honestly instead of failing later at request time.
Open Notebook had already failed once — a four-hour Compose attempt aborted after 18 iterations, ending in a 502. The cause was architectural, not a config mistake: three services shared one Tailscale sidecar's network namespace, and every consumer bound the sidecar's ephemeral container ID. Any sidecar restart orphaned all of them.
dbStatus: online and data intact.
Real migration experience — seven production services with live data, not a
tutorial cluster.
Pattern recognition — services deliberately chosen to de-risk distinct
migration shapes before critical ones were touched.
Risk management — parallel running, verification-before-decommission,
instant rollback, Retain volumes.
Honest documentation — the failures are published, including one that
appeared to lose data.
Modern private-cluster networking — how real corporate meshes work, with
zero public attack surface.
Cost engineering — production-grade patterns at $0/month.