← Back to case study · Agent Team

Agent Team — Plan & Integrations

Two companion documents describing how to extend Hermes with delegate_task sub-agents, surface them via a new k3s agent-team service, and wire LiteLLM + Vaultwarden for guardrails and scoped secrets.

Author: Hermes Agent Date: 2026-09-29 Status: sketch — needs approval Companion to: case-study
Itinerary — five-phase plan for the skeleton

**Author:** Hermes Agent **Date:** 2026-09-27 **Status:** Sketch — needs your approval before any step runs. **Standing rule (recurring from earlier this week):** Don't add infra whose first use-case you can't name. This plan is intentionally small and gated on that question.

Itinerary: Extend Hermes with delegate_task + k3s as the agent-team control plane

Author: Hermes Agent

Date: 2026-09-27

Status: Sketch — needs your approval before any step runs.

Standing rule (recurring from earlier this week): Don't add infra whose first use-case you can't name. This plan is intentionally small and gated on that question.


What this plan is and isn't

It is:

  • A small extension to Hermes itself (which you already run) using delegate_task for in-process sub-agents.
  • A new k3s service — agent-team — that exists only as an observation surface. It exposes a single tailnet ingress that shows you (the operator) what the agents are doing, what state they are in, and what they are about to do.
  • A pattern where the Tailscale Kubernetes operator (already running) does the control-plane work: issuing the device certificate, registering the node, enforcing the tag:container ACL, terminating TLS at the tailnet edge.

It is not:

  • A new vendor (no OpenHands, no Paperclip, no Composio, no Grok sub-agents).
  • A new "control-plane" product. The Tailscale operator + k3s are the control-plane. We are just adding one ingress that rides them.
  • A replacement for anything that exists today. It composes with what you have.

Why these two pieces compose

The plan rests on a single observation: sub-agents need three things, and each of your existing components already provides one:

Sub-agent need Existing component What it does
A way to spawn child work Hermes delegate_task (already enabled, verified) Spawns an in-process sub-agent; returns its final summary; lives in the same memory + skill context as the parent
A way to surface "I am alive and doing X" to the operator k3s + Tailscale operator (already running) Hosts the agent-team ingress on the tailnet; TLS terminates at the Tailscale edge; ACL controls who can read
A way to record state across runs ~/.hermes/cache/jobs/<id>.jsonl (already used by cron) + LiteLLM proxy (already running in litellm ns) Sub-agents can write to the existing job-dir convention; LiteLLM logs every model call with cost+latency; both are tailnet-only

That's the whole architecture. There is no missing piece.


The five-stop itinerary

Stop 0 — Naming the first use-case (gating step)

Before any code lands, name the actual work. Three example shapes; pick one or name a different one:

Shape Concrete job What "sub-agent" would do
A. Triage + escalate Every morning, scan the SkillSpector weekly report, the Litellm error log, and the k3s ingress health, and produce a one-paragraph "morning brief" with action items. One parent (Hermes main) spawns 3 sub-agents, one per source. Each sub-agent reads its assigned log, classifies findings, and returns a 2-line summary. Parent stitches the three summaries into the brief.
B. Codebase maintenance Every Friday, scan the case-study-site repo for broken links, missed milestones, and outdated status. One parent spawns N sub-agents (one per markdown file). Each sub-agent reads its file, returns "OK" / "stale" / "broken". Parent rolls up.
C. Incident response When a watchdog cron fails twice in a row, dispatch an investigation. Parent spawns one sub-agent with a focused prompt and a time budget. Sub-agent returns a structured `diagnosis recommendation next-step` JSON. Parent posts to Telegram with a confidence score.

Why this matters: delegate_task is stateless per-call. Each sub-agent gets a fresh context window, runs to completion, returns its summary. If the first use-case is vague ("manage my infrastructure"), the sub-agent returns garbage. If it's specific ("read this file, classify these 5 categories"), it returns gold. The plan is gated on the user (you) naming shape A, B, C, or another specific one — not on building tooling first.


Stop 1 — Set up the agent-team namespace + service

A new namespace agent-team, hosting one service:

k3s namespace:  agent-team
deployment:     agent-team (single pod, image = hermes-agent-ctl:latest)
service:        agent-team (ClusterIP, port 8080)
ingress:        agent-team-ingress (ingressClassName: tailscale)
                hostname: agent-team-agent-team-ingress.equalacorn-minnow.ts.net

What agent-team actually is: a tiny HTTP server (≤200 lines of Python or Go) that:

  1. On startup, registers itself with tailscaled on the host (via the operator's API).
  2. Exposes GET /api/state — returns the current snapshot of every active sub-agent run: {run_id, parent_id, started_at, status, last_summary}.
  3. Exposes GET /api/recent?n=20 — returns the last N completed sub-agent runs from ~/.hermes/cache/jobs/.
  4. Exposes GET /api/health — standard k8s probe.
  5. Optionally exposes a read-only WebSocket stream at /api/live for a future observability UI.

Why a separate service instead of folding it into Hermes:

  • Decouples "the agent that does work" from "the surface you watch work on."
  • Lets the agent-team pod restart (and roll new images) without interrupting any in-flight sub-agent run.
  • Lets you put the agent-team namespace under a stricter RBAC + network policy than the rest of the cluster.

What it inherits for free from your stack:

  • TLS at the Tailscale edge — no cert management.
  • tag:container ACL — same protection as Vaultwarden, Open Notebook, etc.
  • k3s service mesh — agent-team can read state from other namespaces via the existing tailnet IP, e.g. talk to LiteLLM at litellm-litellm-gateway-ingress.equalacorn-minnow.ts.net:4000 if needed.
  • Backup discipline — k3s services are excluded from hermes-dr-sync.sh; agent-team state should live in ~/k3s-data/agent-team/ and be added to vaultwarden-backup.sh-style cron (deferred — see stop 5).

Stop 2 — Wire delegate_task to publish state

Sub-agents already exist via delegate_task. The new bit is they publish to agent-team on every transition:

Sub-agent event HTTP call to agent-team
Started POST /api/runs { run_id, parent_id, prompt_hash, started_at }
Heartbeat (every 30s while running) PATCH /api/runs/{run_id} { last_heartbeat }
Completed PUT /api/runs/{run_id} { status: "ok", summary }
Failed PUT /api/runs/{run_id} { status: "error", summary, error }

Implementation:

  • A small agent_team_client.py (≤100 lines) lives in ~/.hermes/career-levelup-javid/projects/agent-team/.
  • A Hermes skill agent-team is added: ~/.hermes/skills/agent-team/SKILL.md.
  • The skill teaches sub-agents to call the client on every state transition.
  • The client uses HTTP, not a Kafka-style queue — every event is a single round-trip; a missed event = a stale row, not data loss (because the ~/.hermes/cache/jobs/<id>.jsonl log is the source of truth and the agent-team service is just an index of it).

Why HTTP and not something heavier: sub-agents run inside Hermes's own process. They inherit Hermes's environment, including its tailnet access. A direct HTTP call to http://agent-team.agent-team.svc.cluster.local:8080/api/runs is the simplest thing that works, and the k3s DNS does the rest.


Stop 3 — The "operator dashboard" — what you actually look at

This is the bit I want to think about before we build it. Three options, ranked by what I'd recommend:

Option What you see Cost
i. Just curl + Telegram A cron job on agent-team posts a "last 24h summary" to your existing Telegram chat (the one SkillSpector Weekly already uses, ID 8590652454). Zero new UI. ~50 lines of Python.
ii. A single static HTML page A GET / returns a minimal page with one table: "active runs" + "last 20 completed" + "last error". Reads from the same JSONL log. ~150 lines of HTML+JS. Looks like a 2005-era index.html table, which is the right amount of work for this surface.
iii. A Grafana panel Add the agent-team pod as a Prometheus scrape target; build a small dashboard. Real work, real polish. Probably overkill for v1.

Recommendation: ship (i) now, add (ii) only if you actually look at the cron output more than once a week. The Telegram digest is the smallest unit that delivers the value (you see it every morning). Anything more is yak-shaving until you know you want it.


Stop 4 — Permissions + ACL

Sub-agents will run with the same permissions as the parent Hermes process. That means:

  • The sub-agent can do anything the parent can do (read ~/.hermes/, write ~/hermes-backup/, tailnet access, k8s access if parent has it).
  • A misbehaving sub-agent has the same blast radius as a misbehaving parent.

Three ways to handle this, in order of effort:

Approach What it adds Effort
Trust + audit Trust the sub-agent prompts (parent crafts them carefully). Audit via the agent-team state log. None now.
Tool scoping Limit delegate_task children to a specific tool allowlist (no terminal tool, only read_file + search_files). ~30 lines in agent_team_client.py. Reduces blast radius from "anything Hermes can do" to "read-only filesystem access."
Per-run RBAC Spawn the sub-agent in a separate process with a different uid + a different ACL tag. Real work, requires Linux namespaces or a separate container. Skip for v1.

Recommendation: ship Trust + audit, add Tool scoping as a v1.1 if you ever spawn a sub-agent that touches anything destructive. The named-use-case gate (Stop 0) is the real protection — narrow prompts produce narrow behaviors.


Stop 5 — Backups

The new state surfaces and their backup story:

Surface Lives in Backed up by Risk if lost
Sub-agent run history ~/.hermes/cache/jobs/*.jsonl Already in hermes-dr-sync.sh (whole-~/.hermes sweep with exclusions) None — agent-team rebuilds from these
agent-team namespace k8s manifests ~/.hermes/career-levelup-javid/projects/agent-team/ Need to add to ~/hermes-backup mirror Re-deployable from the repo; low risk
Any persistent state inside agent-team pod ~/k3s-data/agent-team/ Need a cron (mirror vaultwarden-backup.sh) Last-N runs cache; regenerable from JSONL log
TLS cert Tailscale-managed Tailscale None

Backup step: ship the ~/.hermes/scripts/agent-team-backup.sh cron alongside the rest, with the same gzip -n, integrity-gate, and content-hash no-op discipline you wrote for Vaultwarden. Defer until Stop 1 actually lands — premature backup-of-nothing is real yak-shaving.


The order of operations (the actual itinerary)

Phase Steps Estimated time What blocks it What it unblocks
Phase 0 — Pick a use-case You (Javid) name a shape A/B/C from Stop 0, or a different one. 5 min (a single decision) Nothing. Everything below.
Phase 1 — Skeleton service 1. Create agent-team namespace.<br>2. Write agent-team deployment + service + ingress manifests.<br>3. Write the minimal agent_team_server.py (/api/health, /api/runs POST/PUT).<br>4. kubectl apply -f + verify ingress 200. 1–2 hours. Phase 0. Phase 2.
Phase 2 — delegate_task wiring 1. Write agent_team_client.py.<br>2. Write the agent-team skill (SKILL.md).<br>3. Run one end-to-end test: parent spawns a no-op sub-agent → sub-agent calls POST /api/runs → agent-team records it → you see it on the ingress. 1 hour. Phase 1. Phase 3.
Phase 3 — Telegram digest 1. Write the daily-summary cron that posts to telegram:8590652454.<br>2. Run for a week, observe, tune the digest. 1 hour. Phase 2. Phase 4.
Phase 4 — Real first use-case Build the actual sub-agent for the use-case from Phase 0. This is the only phase whose duration is open-ended, because it depends on the job. Depends. Phase 3. Phase 5.
Phase 5 — Backup cron + DR 1. Write agent-team-backup.sh.<br>2. Wire into cron, add to ~/.hermes/cron/jobs.json, re-pin model.<br>3. Document restore in ~/vault/30-systems/disaster-recovery.md. 30 min. Phase 4. Done.

Total wall-clock if the use-case is shape A or B: ~5 hours across a week. Most of that is verification, not code.


What this plan does NOT cover (so we don't get scope creep)

  • Multi-host sub-agents. Sub-agents run inside the Hermes process. They don't span hosts. If you ever need that, the answer is OpenHands Agent Canvas or a future k3s-native agent runtime. Out of scope.
  • Cross-cluster dispatch. This box is the cluster. Server-B is reachable on the network but not part of k3s. Sub-agents cannot dispatch to server-B without first unblocking the ACL (the parked thread). Out of scope.
  • A marketplace for agents. The "sub-agents are prompt patterns" model. The PaperClip compose draft in scratch could be revived for this, but it's a different feature. Out of scope.
  • A real UI. Stop 3 option (ii). Defer until you actually want it.
  • Laya integration. Laya is parked per last session. If/when Laya becomes a thing, the agent-team state log is exactly the data Laya would fine-tune against, but that's a separate decision.

The honest decision points

Three things I want you to push back on before we start:

  1. Phase 0 — name the use-case. A, B, C, or something else. Without this, Phase 4 is unbuildable.
  1. The "operator dashboard" choice (Stop 3). I recommend (i) Telegram digest first; (ii) static HTML only if you actually use (i) for two weeks.
  1. Permissions model (Stop 4). Trust + audit for v1, tool-scoping as v1.1 only if you ever spawn a sub-agent that touches a destructive tool. Worth confirming you'd actually notice a misbehaving sub-agent — the digest + state log are the audit trail.

What lands on disk if you say yes

File Purpose
~/.hermes/career-levelup-javid/projects/agent-team/namespace.yaml k8s namespace declaration
~/.hermes/career-levelup-javid/projects/agent-team/deployment.yaml Deployment + Service + Ingress
~/.hermes/career-levelup-javid/projects/agent-team/agent_team_server.py The HTTP observation surface
~/.hermes/skills/agent-team/SKILL.md The skill that teaches sub-agents to publish state
~/.hermes/skills/agent-team/scripts/agent_team_client.py The client sub-agents call
~/.hermes/skills/agent-team/references/api.md Endpoint contract
~/vault/30-systems/agent-team.md Operational doc, restoration procedure, ACL notes
~/.hermes/scripts/agent-team-backup.sh Daily backup, deferred to Phase 5
~/hermes-backup/skills/agent-team/... Auto-mirrored by DR sync

Plus a new cron job in ~/.hermes/cron/jobs.json for the daily Telegram digest. Pin to minimax-m3 like everything else.


Why I'm not just shipping this

Because you said earlier this week: "stop, lay out what would actually be worth building." That pattern is the right thing to honor here. The plan is conservative on infra and aggressive on use-case clarity. Phase 0 is the entire plan's hinge — if you can't name the work, the agent-team namespace is another Vaultwarden-style artifact that does nothing.

If you want me to start at Phase 1 with a placeholder use-case (something harmless like "summarize the last 10 cron outputs") and revisit the real use-case at Phase 4, that's also a valid path. Otherwise, name the use-case first.


2026-09-29 update: codebase-maintainer shipped

The itinerary's "name the use-case" gate was passed with codebase-maintainer.

Phase 1 / Step 1 — observation surface (DONE 2026-09-29):

  • k3s namespace agent-team live, pod Running, Tailscale ingress
  • SQLite WAL observation DB on hostPath PV
  • Dashboard at https://agent-team-agent-team-ingress.equalacorn-minnow.ts.net/

Phase 1 / Step 2 — sub-agent wiring (DONE 2026-09-29):

  • scripts/agent_team_client.py — stdlib client (start/heartbeat/complete/state)
  • scripts/codebase_maintainer.py — first real sub-agent, 9 hygiene checks
  • ~/.local/bin/agent-team-client — shell wrapper for cron prompts
  • SkillSpector Weekly cron (2859e0be8b5e) wrapped with STEP 0 + STEP 4
  • LiteLLM master-key path verified end-to-end (model minimax-m3 answers)

First run R-d2b61a7eab8c (2026-09-29 19:57:20 UTC, 8s duration):

  • 3 owned repos audited (hermes-cron-backup, hermes-backup, geopolitics-command-center)
  • 9 skipped (upstream/vendor)
  • 18 OK, 9 LOW, 0 MED, 0 HIGH
  • LLM triage correctly identified the deepresearch/ uncommitted file and

the 1.7MB skillspector-baseline log artifact by name

  • Run visible in /api/state with full event timeline (started → completed)

Status of the itinerary's 3 design shapes:

  • Shape A (LiteLLM-only) — wired via master key, no virtual key needed yet
  • Shape B (Vaultwarden org + collections) — deferred (no role needs vault access yet)
  • Shape C (full audit trail with run_id everywhere) — deferred (Shape A sufficient)

Next validation: SkillSpector Weekly cron tick at 2026-10-06 06:00.

If the wrap fires correctly, the entire agent-team pattern is proven.

Integrations — guardrails + secrets sharing (LiteLLM & Vaultwarden)

**Author:** Hermes Agent **Date:** 2026-09-27 **Status:** Sketch — needs your approval before any step runs. **Companion to:** `agent-team-itinerary.md` (the skeleton plan). This document is independent — you can ship the skeleton without this, or ship this without the skeleton, but they're most...

Guardrails + Secrets Sharing: Wiring agent-team with LiteLLM and Vaultwarden

Author: Hermes Agent

Date: 2026-09-27

Status: Sketch — needs your approval before any step runs.

Companion to: agent-team-itinerary.md (the skeleton plan). This document is independent — you can ship the skeleton without this, or ship this without the skeleton, but they're most powerful together.


What this document is

The agent-team skeleton (separate doc) gives you a place to watch sub-agents. This document gives you the guardrails and secrets infrastructure so the sub-agents can do real work safely — rate-limited, budgeted, with credentials scoped per role, with audit trails.

It addresses one specific question: when a sub-agent runs, what is allowed to happen, and how is the policy enforced?


Why this matters more than the skeleton

A sub-agent without guardrails is a delegate_task that can do anything the parent Hermes can do. That includes:

  • Reading ~/.hermes/.env (and every API key you've ever put in there).
  • Calling LiteLLM with your master virtual key (unlimited spend).
  • Sending arbitrary network requests over the tailnet (touching Vaultwarden, gitea, GitHub API, etc.).
  • Writing to ~/hermes-backup/, ~/vault/, ~/.hermes/skills/, anywhere.

Without guardrails, a misbehaving sub-agent has the same blast radius as a misbehaving parent. That's the security model that fails the audit. The shape of "guardrails + secrets" is what changes that.


Three integration shapes, ranked

Shape What gets wired Effort When it's right
A. LiteLLM-only Sub-agent model calls route through LiteLLM with a per-role virtual key. Vaultwarden not involved. Low (~30 min) Triage / read-only use cases; low-blast-radius
B. LiteLLM + Vaultwarden As A, plus sub-agents fetch their own credentials from Vaultwarden on demand, scoped per role. Medium (~2 hr) Codebase maintenance, multi-service work; medium-blast-radius
C. A + B + agent-team policy gate As B, plus agent-team is consulted before every model call and every vault read; run_id flows through everything for full audit trail. High (~6 hr) Incident response, financial-sensitive work; production-grade

My recommendation: ship B for v1, plan C as v1.1. A is too thin (the "sub-agent inherits parent's .env" problem stays). C is the right answer eventually, but it's not free.


Shape A: LiteLLM-only (the lightweight version)

What it is

A sub-agent's model calls flow through LiteLLM with a virtual key tied to the sub-agent's role, not to Hermes. LiteLLM's existing per-key rate-limit and cost-tracking features do the enforcement.

What gets created

~/.hermes/.env
├── LITELLM_BASE_URL              # existing, points at litellm-litellm-gateway-ingress.equalacorn-minnow.ts.net:4000
├── LITELLM_MASTER_KEY            # existing, parent's key — DO NOT give this to sub-agents
├── LITELLM_AGENT_TEAM_KEY        # NEW — virtual key, scoped to "agent-team" team in LiteLLM
├── LITELLM_AGENT_TEAM_TRIAGE_KEY # NEW — virtual key for the triage role
└── LITELLM_AGENT_TEAM_CODEBASE_KEY # NEW — virtual key for the codebase role

Each virtual key has:

  • A model allowlist (e.g. triage can only call minimax-m3; codebase can call minimax-m3 + deepseek-v4-flash).
  • A rate limit (e.g. 60 requests/minute, 100k tokens/hour).
  • A daily budget (e.g. $0.50/day; auto-pause on breach).
  • An alias that propagates to LiteLLM logs (so you can grep "what did the triage role do today").

What the parent does

When Hermes's parent process spawns a sub-agent via delegate_task:

# in agent_team_client.py
SUB_AGENT_KEYS = {
    "triage":     os.environ["LITELLM_AGENT_TEAM_TRIAGE_KEY"],
    "codebase":   os.environ["LITELLM_AGENT_TEAM_CODEBASE_KEY"],
}

# sub-agent receives a role-named environment variable
role = parent_prompt.metadata["role"]  # e.g. "triage"
sub_agent_env["LITELLM_API_KEY"] = SUB_AGENT_KEYS[role]

The sub-agent uses the role's key instead of inheriting LITELLM_MASTER_KEY. The parent's master key stays out of the sub-agent's environment.

What changes on disk

File Change
~/.hermes/.env 3 new vars (LITELLM_AGENT_TEAM_KEY, LITELLM_AGENT_TEAM_TRIAGE_KEY, LITELLM_AGENT_TEAM_CODEBASE_KEY)
LiteLLM admin UI Create the team + virtual keys with the allowlist/limits/budgets
agent-team/SKILL.md Document which role gets which key
agent_team_client.py Inject the right key based on role

Time estimate: ~30 minutes.


Shape B: LiteLLM + Vaultwarden for sub-agent secrets

What it is

Shape A, plus sub-agents fetch their own credentials from Vaultwarden on demand, scoped to the role. The parent never has the sub-agent's role-specific credentials in its .env. The parent only has the Vaultwarden user token, which is the pointer to where the credentials live, not the credentials themselves.

Why this matters

If the parent Hermes's .env is compromised (or if a sub-agent somehow reads it), the attacker gets:

  • LiteLLM master key → unlimited model spend.
  • Vaultwarden admin token → access to every secret in the vault.
  • Cloudflare token → can deploy to your Pages project.
  • GitHub token → can push to repos.
  • etc.

If we move role-specific credentials into Vaultwarden collections and the parent only holds the user token + org/collection IDs, the blast radius shrinks. The parent's .env becomes "credentials Hermes itself needs" + "pointers to where sub-agent credentials live."

What changes in Vaultwarden

Right now you have 1 user. Sub-agents are not users. To give them scoped credentials, you have three options:

Approach Effort Verdict
Open registration, create per-role users 1 min ❌ Contradicts your "registration closed" rule
Create an Organization + collections 20 min ✅ The right answer — share a collection with your existing user, scope access by collection
Use admin-token direct API 30 min ⚠️ Works but bypasses Bitwarden's collection model; every read is "as admin"

Recommended: Organization + collections. This is the Bitwarden-native pattern and it composes with the Bitwarden clients you already use.

The vault shape becomes:

Vaultwarden
└── User: edm***@gmail.com (you — unchanged)
    └── Organization: agent-team (NEW)
        └── Collection: agent-team-triage       (shared with: you)
        │   ├── TAVILY_API_KEY         (NEW)
        │   ├── GITHUB_TOKEN_TRIAGE    (NEW, scoped read-only)
        │   └── ...
        ├── Collection: agent-team-codebase     (shared with: you)
        │   ├── CLOUDFLARE_TOKEN_CODEBASE (NEW, scoped to one Pages project)
        │   ├── GITHUB_TOKEN_CODEBASE   (NEW, scoped to one repo)
        │   └── ...
        └── Collection: agent-team-incident     (shared with: you, future)
            └── ...

What changes on disk

File Change
Vaultwarden admin Create org agent-team, create per-role collections, move/place credentials
~/.hermes/.env Replace direct credentials with Vaultwarden pointers: VAULTWARDEN_URL, VAULTWARDEN_USER_TOKEN, VAULTWARDEN_ORG_ID, VAULTWARDEN_COLLECTION_TRIAGE_ID, VAULTWARDEN_COLLECTION_CODEBASE_ID
agent_team_client.py New module: vault_client.py — fetches secrets from Vaultwarden, caches in memory, revokes on run completion
agent-team/SKILL.md Document which secrets each role can access

Time estimate: ~2 hours.

The .env taxonomy decision

When you move secrets to Vaultwarden, you have to choose for each existing secret:

Secret Move to Vaultwarden? Why
LITELLM_MASTER_KEY No — parent uses this directly Parent's own key
LITELLM_AGENT_TEAM_*_KEY No — parent's env only Sub-agent gets it via injection, not vault
TAILSCALE_AUTHKEY No — already low-privilege Used once at startup
CF_PAGES_TOKEN Yes, into agent-team-codebase Sub-agent may need to deploy
GITHUB_TOKEN Maybe — depends on scope If it's a wide-scope PAT, scope it down then move it
COMPOSIO_* Yes, into relevant role collection Per-role credential
VAULTWARDEN_USER_TOKEN No — but keep in .env with mode 600 The pointer, not a secret in itself
Backup encryption keys No Not used by sub-agents
Telegram bot token No Parent's, not sub-agent's

This taxonomy is the part you need to make a decision on. I'll write a worksheet you can use, but the actual "which secret goes where" requires your judgement.


Shape C: Full audit trail via agent-team as policy gate

What it is

Shape B, plus agent-team is consulted before every model call and every vault read. Every event gets a run_id that flows through every layer of the system.

parent (Hermes)
  ↓ spawns sub-agent with run_id=R-1234, role=triage
  ↓
sub-agent
  ↓ wants to call LiteLLM
  ↓ POST agent-team/api/check { run_id, action: "model_call", model: "minimax-m3", est_tokens: 800 }
  ↓ → 200 OK { allowed: true, remaining_budget: 0.34 }
  ↓ makes the call
  ↓ POST agent-team/api/record { run_id, action: "model_call", cost: 0.012, duration_ms: 2400 }
  ↓
sub-agent
  ↓ wants to fetch TAVILY_API_KEY from vault
  ↓ POST agent-team/api/check { run_id, action: "vault_read", secret_name: "TAVILY_API_KEY" }
  ↓ → 200 OK { allowed: true, vault_collection: "agent-team-triage" }
  ↓ fetches from vault
  ↓ POST agent-team/api/record { run_id, action: "vault_read", secret_name: "TAVILY_API_KEY", duration_ms: 240 }

This gives you a single source of truth: "run R-1234 did X model calls totaling $0.34, fetched 2 vault secrets, took 12 minutes, returned summary Y."

What gets built

  • agent-team service grows from a 200-line observation surface to a 600-line policy gate.
  • Every sub-agent call path (LiteLLM client, Vaultwarden client) gets a check + record wrapper.
  • A new agent-team/api/runs/{run_id}/audit endpoint returns the full timeline.

When it's right

When a misbehaving sub-agent would cost you real money, real data, or real user trust. For Triage (read-only, no destructive ops), it's overkill. For Incident Response (financial / sensitive), it's the only honest answer.

Time estimate: ~6 hours.


What I'd want you to confirm before any wiring

Three decisions, in priority order:

1. Which shape? A, B, or C. My recommendation is B for v1, C as a v1.1. But you might say "A is enough for now" because you don't have a real use-case yet, and that's the right answer too.

2. Vaultwarden organizational model. The recommended path is "create an Organization called agent-team and use collections." This changes the vault's data model — you'll see an org in the Bitwarden clients that wasn't there before. Confirm you're OK with that, or push back.

3. Role taxonomy. How many distinct sub-agent roles do you foresee? One (just triage for now), two (triage + codebase), or more? Each role gets:

  • A LiteLLM virtual key
  • A Vaultwarden collection
  • An entry in the agent-team policy table

If you don't know yet, ship one role (agent-team/role:triage) and clone the table later. This is the same "minimal first" stance as the skeleton itinerary.


What does NOT change

  • Sub-agents still run inside Hermes's process. No new VMs, no new vendor products.
  • Tailscale + k3s operator still handle the network. No public endpoints.
  • Vaultwarden stays as your secrets store. We extend it with an org, not replace it.
  • LiteLLM stays as your model gateway. We add virtual keys, not a new layer.
  • agent-team (Phase 1 of the skeleton) becomes a richer surface, but if you decide not to build it, Shape A and Shape B still work — they just don't have the observation UI.

File list if you say yes to Shape B

File Purpose
~/.hermes/career-levelup-javid/projects/agent-team/integration/litellm-virtual-keys.md How to provision the virtual keys (admin UI steps + curl fallbacks)
~/.hermes/skills/agent-team/scripts/vault_client.py The Vaultwarden client sub-agents call
~/.hermes/skills/agent-team/references/vault-secrets-catalog.md The taxonomy: which secret goes where
~/vault/30-systems/agent-team-integrations.md Operational doc: how to add a new role, how to rotate a role-specific secret
~/vault/30-systems/vaultwarden.md Update — add the agent-team org section
~/hermes-backup/skills/agent-team/... Auto-mirrored by DR sync

Plus changes to:

  • ~/.hermes/.env (per role above)
  • ~/.hermes/.gitignore (exclude new vault pointers if they're already in any tracked file)
  • LiteLLM admin config (the virtual keys + budgets)

The honest trade-off summary

Shape A Shape B Shape C
Mishandled credential blast radius Sub-agent reads parent's .env once compromised = everything in .env is gone Sub-agent reads parent's .env = only LiteLLM master + Vaultwarden pointer, not the per-role credentials Same as B, plus every action is auditable
Audit trail LiteLLM logs only LiteLLM + Vaultwarden access logs Single unified log in agent-team
Effort 30 min 2 hr 6 hr
Maintenance One-time, then stable Vaultwarden collection hygiene (when you add a role, you add a collection) agent-team grows over time as policies refine

The right answer isn't "always C." It's "B is the floor for any production-like workload, C is the ceiling for any sensitive workload." For Triage (shape A from the itinerary's Stop 0), A is defensible. For Codebase (shape B), B is the floor. For Incident (shape C), C is mandatory.


Cross-document note

This document and agent-team-itinerary.md are independent. You can ship either without the other:

  • Ship the itinerary's Phase 1–3 without this document → you have sub-agents with no guardrails.
  • Ship Shape A from this document without the itinerary → you have guardrails on the model side, but no observation surface.
  • Ship Shape B alone → you have per-role credentials but no agent-team to watch them.

The full system is both documents together. Your move on which pieces you want first.


2026-09-29 update: Shape A wired end-to-end

When I went to build Shape A's "LiteLLM virtual key per sub-agent role," I

discovered nothing actually calls LiteLLM today:

  • Parent Hermes uses ollama-cloud directly (model minimax-m3)
  • All 12 cron jobs route to ollama-cloud, not LiteLLM
  • No MCP server for LiteLLM configured

So I deferred the virtual key path (per your "don't add infra whose first

use-case you can't name" rule) and went with master-key for the first

sub-agent. Worked first try: codebase-maintainer registered itself in

agent-team, called LiteLLM with the master key, got a real triage from

minimax-m3, then completed itself.

Shape B (Vaultwarden org/collections) remains deferred — the first

sub-agent (codebase-maintainer) reads files but doesn't need vault secrets.

When a role that calls external APIs lands, that's when Shape B becomes

useful.

Shape C (full audit trail with run_id flowing everywhere) — partial

implementation already:

  • run_id flows into agent-team (full event timeline)
  • run_id flows into the LiteLLM call via the model call (LiteLLM's logs

show it as a request from the timestamp the sub-agent ran)

  • run_id does NOT yet flow into Vaultwarden (no vault calls)

When Shape B becomes real, Shape C becomes cheap.