claude-projects/agent-builder/agent_sudo_design_decisions.md

# Agent-Sudo (projects DB id=176) — Design Grill-Me: Locked Decisions & Open Gaps

**Status:** ✅ GRILL-ME COMPLETE (2026-07-08, Opus 4.8/high). All decisions D1–D10 + gap-#5 cross-cutting LOCKED. Phased build prompts P0–P5 written to `agent_prompts.md`. NEXT = user review of prompts, then dispatch (not yet launched).
**Started:** 2026-07-08 (Opus 4.8 / high). Design doc for the pre-existing id=176 design; this file records the sub-design decisions the DB row left unspecified.

## Verified current state (checked live 2026-07-08)
- ❌ No Incus/LXC/LXD installed on server-01 (192.168.1.90) — P1 installs it
- ❌ No `command_audit` table in `projects` DB (table `infrastructure_projects` holds id=176) — P0 creates it
- ✅ `agent_test_results` table EXISTS in `api_business` DB (verified 2026-07-08); `automation_ideas` + `behavior_changes` also in `api_business` (NOT the empty `automation_ideas` DB)
- ✅ timeshift installed on server-01 (`/usr/bin/timeshift`)
- ✅ **Coolify RETIRED** (control plane stopped 2026-06-25) — deploy Agent-Sudo via plain docker-compose + Jenkins/Gitea, never Coolify; old bridge decommission (P5) = plain stop+rm, nothing to deregister. `coolify-proxy` = standalone Traefik (name kept for cloudflared). See [[project-coolify-traefik-retirement]].
- ✅ **N8N is NOT a caller of the bridge** — all 12 production workflows checked 2026-07-08, zero reference sudo-bridge (they use the now-dead Coolify API + Vault + bitwarden-bridge). So the D8 cutover shadow-diff has ZERO N8N surface; real callers = claude-code / background-agents / (future) Hermes via PROXY_CALLERS.
- ⚠️ **PORT CORRECTION to D8:** Agent-Sudo parallel port = **8084**, NOT 8083 (8083 is bitwarden-bridge). 8084/8085/8086/8087 free on primary as of 2026-07-08.
- Existing sudo-bridge: primary `sudo-bridge-drbjegv07256ki2lpfyr00n8` (port 8082 @192.168.1.88); server-01 bridge (Coolify UUID o2kz1puml1mmneyiqd96mouj, port 8082 @192.168.1.90); shared image `gitea.local/backtalk6858/sudo-bridge:latest`; API keys in Vault `secret/data/sudo-bridge` and `secret/data/sudo-bridge-server01`.

## LOCKED DECISIONS

### D1 — LXC mirror fidelity = COMMAND-SURFACE mirror
Same OS + package versions, same directory structure/paths, same users/sudoers/groups.
**No** live data, **no** running containers. Static, script-rebuildable, zero production-data exposure in sandbox.
(Rejected: state-replica mirror — constant drift, high upkeep, leaks prod data.)

### D2 — Tier-2 semantics = RE-TIER BY REPRODUCIBILITY
The current 4-level testing methodology tests *built automations*; sandbox-testing *arbitrary sudo commands* universally is impractical (would need the rejected faithful replica). So:
- **Reproducible ops** (file edits, package installs, permission/config changes) → run the **sandbox-execute** path in the command-surface mirror, using **static/dry-run validation** (parse, path resolution, forbidden-path check, `--dry-run`/`shellcheck`). Does not claim to prove live runtime behavior.
- **Live-service ops** (`docker`/`systemctl` restarts, network changes, live DB) → **skip the sandbox** (it structurally can't test them) → **Tier 3**: Timeshift snapshot → execute on target → full audit trail. If it fails, restore from snapshot and post-mortem from the audit record.
- Net: the LXC mirror earns its keep only for reproducible ops; live-service safety = snapshot + audit, not a theatrical sandbox run.

### D3 — LOCKED (confirmed 2026-07-08): rollback is scoped, not Timeshift; primary is stricter
Emerged from the "primary has no failover" question. Confirmed by user.
**Tier-4-on-primary = HARD REFUSE, interactive-session-only (option a).** No remote/NTFY approval path for primary Tier-4 — a command dangerous enough to be Tier-4 on the most critical host requires a human at a real terminal. (server-01 may still allow refuse-and-notify at its own lower tiers.)
**Long-term (Horizon B / redundancy vision):** once a redundant failover setup exists (Proxmox mirror + server redundancy/k3s — see [[project_server_redundancy_vision]], [[project_proxmox_test_server]]), revisit allowing full Timeshift restore — with failover, actual downtime → ~0%, so the "full restore = total downtime" objection dissolves and the primary posture can relax. Until then: conservative-by-default.
- **Default Tier-3 rollback = scoped command-specific undo**, captured BEFORE execution (file backup+restore of only touched files; package record+remove; container recovery via **Jenkins redeploy**, not system restore). Blast radius = only what the command touched.
- **Timeshift full-restore = emergency last resort ONLY** (OS-level corruption: dpkg/systemd/libs). User-in-the-loop, NTFY-urgent, accept full downtime. Agent-Sudo may TAKE the snapshot but must NEVER auto-restore primary. Rationale: full restore = total downtime for ALL services (incl. Vault/Authelia/secrets-proxy/media) + reboot, AND reverting /var/lib/docker risks stateful DATA LOSS (Postgres).
- **Host-aware tiering:** primary classifier stricter than server-01. Same command may be Tier-3 on server-01 but **Tier-4 (refuse → require interactive session)** on primary — especially anything touching critical security infra or stateful DB volumes.
- Long-term real fix = Proxmox test-mirror + server redundancy/k3s failover (already in vision). Until then primary posture = conservative-by-default.
- Feeds SUDO.md design (gap #1): the classifier must be **host-aware** (per-host tier overrides).

### D4 — LOCKED (2026-07-08): tier model has NO human-approval gate
The current sudo-bridge tier-2 "NTFY approve/deny then execute" gate is being **eliminated**, not preserved. Rationale (why — this is the whole point of Agent-Sudo):
- It didn't work: user misses NTFY notifications → command times out (900s APPROVAL_TIMEOUT), compounded by up to 5 parallel background agents competing for attention.
- It was theatrical: user was asked to approve commands they couldn't evaluate for safety → rubber-stamping, not a real gate.
- Real safety = testing/verifying the command (sandbox for reproducible ops; snapshot+audit+auto-verify for live ops), NOT a human who lacks context.
- Only human touchpoint remaining = Tier-4 hard-refuse → interactive session. Classifier must be tuned so Tier-4 is a near-zero-frequency event.

**Canonical tier model (safe→dangerous ascending; inverts the live bridge's 0=refuse):**
| Tier | Name | Behavior |
|---|---|---|
| 0 | auto-safe (read-only) | auto-execute, no snapshot |
| 1 | auto-reversible | auto-execute + capture scoped-undo |
| 2 | sandbox-first (reproducible ops) | dry-run/validate in LXC mirror → execute → audit |
| 3 | snapshot+execute (live-service ops) | scoped-undo (+Timeshift snapshot) → execute → full audit → auto-verify |
| 4 | hard-refuse | refuse → require interactive session (host-aware: stricter on primary) |

Migration note (P0/P4): existing allowlist.json uses live-bridge numbering (0=refuse,1=auto,2=NTFY-approve). Remap on cutover: live tier-1 → new tier-0/1 (by read vs write); live tier-2 → new tier-2 or 3 (by reproducible vs live-service); danger vetoes → new tier-4.

### D5 — LOCKED (2026-07-08): matching engine + seed corpus + audit-driven self-completing loop
**Engine = option (a):** evolve the existing model — ordered rule list, most-specific-first (first-match-wins), `fnmatch` globs (existing allowlist.json migrates ~1:1), regex danger-veto = auto-Tier-4. Each rule gains optional per-host tier override + verb class. Verb-heuristic is a *fallback default only*, not the primary engine.
**Miss behavior (host-aware, per D3):** allowlist miss on **primary → hard-refuse (Tier-4)** (primary never guesses); on **server-01 → verb-heuristic default** allowed (sandbox-side may guess). Ambiguous even under heuristic → Tier-4.
**Seed corpus:** pre-populate SUDO.md via internet research on the common command set we actually run + what each does → assign each a tier up front. Reduces cold-start Tier-4 misses.
**Audit-driven classification loop (the self-completing part — this is the training-data mechanism):**
1. Unknown command arrives. Reproducible-verb unknowns route to the **sandbox** (Tier-2 treatment) — NOT the live target. Live-service unknowns on primary → Tier-4 until classified (no blind execution on primary).
2. Run in sandbox → analyze the audit trail (exit code, output, stderr).
3. If it executes cleanly → evidence supports a tier; propose the classification.
4. If it fails because the sandbox is missing something (path/package/config/user) → analyze the audit trail to identify the gap → add it to the command-surface mirror **rebuild script** → rebuild → re-run. This makes the LXC mirror (D1) self-completing over time.
5. The AI evaluator (P3) produces a **proposed** tier + audit evidence and writes the rule into SUDO.md in a **`proposed` (inactive) state on BOTH hosts** — a proposed rule cannot be used to execute until reviewed. It fires a **low-priority NTFY** ("new proposed Agent-Sudo rule — load conversation <X> in an interactive session to review"). Rule becomes `active` only after the review discussion. Logged as training data throughout.
   - **Why this does NOT contradict D4:** D4 killed the blocking *execution*-approval gate (timed out, on live commands the user couldn't evaluate). This is a **non-blocking async review of a config/rule change** — nothing is stalled waiting on it (no timeout failure mode), it's low-frequency (meta-level), and it happens in a session where the user CAN evaluate it with Claude's help. Different mechanism, different risk profile.
   - Implementation detail (defer): NTFY must carry a stable conversation pointer for the review session (candidate: a dedicated `agent-sudo-rules` conversation, or the agent-builder conversation).

### D6 — LOCKED (2026-07-08): operating principle = minimize human involvement, not eliminate it
Until human-level AGI exists, reducing the user's involvement to exactly zero is not achievable. Design target for every autonomous system = **minimize** required human touchpoints, not **eliminate** them. Legitimate residual touchpoints (Tier-4 interactive execution; proposed-rule review) are kept deliberately, made non-blocking and low-frequency, and each is justified. (Candidate universal feedback memory — applies beyond Agent-Sudo.)

## OPEN GAPS (resume grill-me here)

### D7 — LOCKED (2026-07-08): LXC architecture on server-01
Verified live: both hosts = LMDE 7 "gigi" (Debian 13 trixie base), kernel 6.12.90; server-01 = 31GB RAM/27GB free, 173GB free disk, btrfs tooling present (no zfs), no LXC/LXD/Incus yet. Primary confirmed CLEAN of all testing infra (no LXC/sandbox/test containers; only prod postgres + vault; timeshift present = intended D3 snapshot tool).
- **Container tech = Incus** (Debian-trixie-native; profiles/limits/instant snapshots for cheap reset-to-baseline).
- **Base image = `images:debian/13` (trixie).** LMDE≈Debian under the hood; LMDE's deltas are desktop/theming, irrelevant to sudo command behavior. Confirmed acceptable by user.
- **Storage = btrfs pool** (loop-file or subvolume) for near-instant snapshot/rollback; fallback `dir` backend if root isn't btrfs.
- **Network = NAT via default `incusbr0`** — outbound works (nala/apt during classification tests), ZERO inbound LAN exposure (sandbox must not be reachable on 192.168.1.x).
- **Limits = 2 vCPU / 2GB RAM / 10GB disk each** (ephemeral scaffolds, no services).
- **Rebuild = one parameterized `build-sandbox.sh <primary|server-01>`** (D1 script-rebuildable; D5 loop appends missing deps). Differs per host only in package set + paths + users/sudoers.
- **P1 prerequisite = read-only host inventory** (the mirror spec): `dpkg --get-selections`, users/uids+groups+sudoers STRUCTURE (never secrets), relevant dir tree (`/opt/appdata`, `/data/coolify`, `/etc/...`). Both mirrors (`sandbox-primary`, `sandbox-server01`) run ON server-01; `sandbox-primary` mirrors production's command surface but never runs on production.

### D9 — LOCKED (2026-07-08): privilege + security model = FULLY AUTONOMOUS, no human arming
Hard constraint (user): must be fully autonomous — if the user is unavailable, automation must not stall. So NO human-arming gate on the execution path. Once full autonomy is required, "can the machine escalate on its own?" = yes (settled); security moves OFF access-gating (theater) ONTO the constraint layer.
**The reframe (the crux):** separate two powers —
- **EXECUTE an already-vetted command** → fully autonomous, no human ever, always available (Tiers 0–3).
- **EXPAND what's allowed** (new command / raise tier / widen allowlist) → the ONLY human touchpoint; rare, async, non-blocking (D5 proposed-rule review). The autonomous system CANNOT widen its own allowlist.
Attacker win = "run arbitrary dangerous command" ⇒ requires widening the allowlist ⇒ impossible autonomously; executor enforces danger-veto regardless. Full compromise of the smart brain ⇒ only inherits "run already-safe/reversible/audited commands" = bounded blast radius.
**Unavailability is NOT a single point of failure:** all Tier 0–3 automation runs with user asleep/away; only (a) activating a brand-new capability and (b) catastrophic Tier-4 wait for a human — both SHOULD wait.
**Privilege mechanism:** dumb root daemon everywhere (auth-agnostic — survives user's fingerprint→privacyIDEA MFA→YubiKey roadmap; never calls sudo/PAM). Both hosts = always-on SCOPED short-TTL AppRole (reuse existing rotate-and-revoke machinery; policy reads ONLY the one Agent-Sudo secret path). Bitwarden + OS-sudo-password both DROPPED from design (Bitwarden was only for the abandoned OS-password pull).
**Autonomous-security stack (all human-free):** least-privilege allowlist · dumb minimal root executor · independent re-validation (already in code) · signed/read-only SUDO.md (Vault transit — allowlist can't be silently widened) · scoped short-TTL AppRole · executor sandboxing (AppArmor/seccomp/systemd) · egress lockdown (executor has no outbound net) · append-only audit shipped off-box · **anomaly circuit-breaker that self-disarms** (watches own command rate/mix/failure spike → auto-stops Tier-3 + alerts — replaces the human panic button) · **dual-service concurrence on Tier-3** (independent second classifier must agree). Primary stays stricter (D3): larger Tier-4 set, catastrophic → refuse-to-human.
**New tool? NO.** All pieces already exist (allowlist+executor, Vault AppRole + transit signing, AppArmor/seccomp/egress free on host). Falco (runtime anomaly detection) / OPA (policy-as-code) = documented FUTURE-ENHANCEMENT candidates only — new tools add attack surface + maintenance for a solo operator.

### D10 — LOCKED (2026-07-08): Hermes as the always-on independent monitor (future layer)
Hermes (id=138, monitoring brain, local model on server-01, no Claude-API cost) is the natural home for the monitoring layers of D9 — layered on AFTER Agent-Sudo v1 ships.
**Security gains:** (1) Hermes IS the intelligent anomaly circuit-breaker/watchdog — reads the append-only audit stream, trips the breaker on anomaly, NTFYs user; being a SEPARATE process from Agent-Sudo = separation of duties (compromising Agent-Sudo ≠ compromising the watchdog). (2) The independent second opinion for dual-service Tier-3 concurrence. (3) Audit drift-detection + integrity monitoring (SUDO.md signature valid? AppArmor loaded? egress rules present? → tamper alert). 
**Usability gains:** (1) triages/batches the two human touchpoints (proposed-rule activation, Tier-4) into digestible evidence-backed decisions. (2) Drives the D5 sandbox-test-and-analyze loop autonomously so unknown-command classification is one-click-confirm by review time. (3) Self-healing: detects failed side effects → triggers scoped-undo or Jenkins redeploy (per project_jenkins_hermes_boundary).
**HARD BOUNDARY:** Hermes is on the MONITORING side only — observe/analyze/alert/trip-breaker/trigger-pre-vetted-recovery. Hermes NEVER gets execution privilege, NEVER widens the allowlist, NEVER runs arbitrary Tier-3+. If Hermes could command the executor freely it becomes the soft underbelly (compromise Hermes = escalate). Keeps automated dual-control intact: Agent-Sudo (has root, dumb about judgment) + Hermes (no root, smart about judgment) check each other.
**Build order:** Agent-Sudo v1 ships with a simple self-contained rule-based circuit-breaker; Hermes later augments/replaces it as the intelligent watchdog. Ties to project_hermes_exploration_day (capability-map after id=138).

## OPEN GAPS (resume grill-me here)
1. ✅ RESOLVED — SUDO.md classifier (see D4 tiers, D5 engine/seed/loop, D6 principle).
2. ✅ RESOLVED — LXC/Incus specifics (see D7).
3. ✅ RESOLVED — Cutover & rollback (D8).
4. ✅ DROPPED — Bitwarden ephemeral pull no longer in design (D9: Vault AppRole replaces it; no OS-password, no Bitwarden).
   [Security model fully resolved: D9 + D10.]

### D8 — LOCKED (2026-07-08): cutover sequence & rollback
Drop-in replacement: response contract stays identical (`POST /exec` Bearer → `{exit_code, output, ...}`; T4 refuse = 403 like today's allowlist-miss). Callers need zero changes; T2/T3 just add latency (sandbox/snapshot) transparently. D4 removes the tier-2 NTFY-approval block entirely.
- **Deploy order = server-01 FIRST, primary LAST** (user confirmed). server-01 = testing host, low blast radius, and hosts the sandbox LXCs (T2 local). Primary only after a clean 4-level pass on server-01.
- **Parallel run via ports:** Agent-Sudo stands up on **new port 8084** (⚠️ corrected 2026-07-08 from 8083 — 8083 is bitwarden-bridge) while old bridge keeps serving **8082** untouched. Shadow-validate (4-level gate on 8083 + replay known-safe commands through both, diff results). Callers stay on 8082 during validation → zero disruption.
- **Atomic swap when green:** flip Agent-Sudo to 8082, demote old bridge to 8083 hot-standby (kept running, not deleted).
- **Rollback = swap ports back** (old bridge one flip from resuming 8082). Retain old bridge container + image **14 days** before decommission (P5).
- **Primary cutover = MANUAL port-swap by user** in an interactive session (user confirmed). Agent-Sudo is built + validated autonomously everywhere; the final "make it live on production" flip on primary is user's hands on keyboard (per D3/D6). server-01 cutover = autonomous.
4. **Bitwarden ephemeral sudo password pull** — confirm path via bitwarden-bridge; never stored at rest.
5. ✅ RESOLVED (2026-07-08) — **Cross-cutting:**
   - **Per-phase rollback/abort** — matrix baked into the P0–P5 prompts (P0 drop-table / P1 incus-delete / P2 delete-repo / P3 revoke-policy / P4 port-flip-back / P5 re-pull-image). No phase is destructive to the OLD bridge until P5.
   - **Vault/N8N/Coolify dry-run paths** — Vault = additive `agent-sudo` policy + revoke-self smoke test (least-priv proof); N8N = ZERO surface (not a caller); Coolify = retired, Agent-Sudo deploys OUTSIDE it via Jenkins, P5 = plain stop+rm.
   - **Training-data logging** — 6 decision points → `command_audit` rows; GOLD label = AI-eval-proposal → human review verdict (`human_verdict`). P0 schema carries the fields.
   - **Blast radius per tier** — 0 minimal→3 medium/large→4 zero; capped on primary by shifting the dangerous middle up to refuse; **Tier-3-on-primary = highest-attention path** (most tests + dual-concurrence + tightest breaker). Full-compromise ⇒ inherits only already-vetted T0–3, cannot widen allowlist or reach T4.
   - **Blocks/unblocks** — id=176 has NO blocker (buildable now); no DB row lists it as a blocker (link is architectural, not recorded). Unblocks: Hermes self-healing (id=138), the sandbox-testing program (it's the first test of the harness), and any Horizon-B autonomous automation needing unattended privileged exec. Does NOT unblock the 12 broken N8N workflows (those need the Jenkins/Coolify-API migration).

## AFTER GRILL-ME COMPLETES
Write the **phased** background-agent prompt SET (NOT one monolith — see reasoning in session):
- P0: DB foundation (`command_audit` in projects + ensure `agent_test_results` in api_business) — design-complete, no server risk.
- P1: LXC/Incus setup + command-surface mirror build on server-01.
- P2: Python service = drop-in sudo-bridge replacement (same API), classifier reads SUDO.md, audit → command_audit.
- P3: Bitwarden ephemeral pull + Timeshift automation + AI evaluator (tier-2/3 flow).
- P4: deploy both servers behind old bridge, run 4-level gate.
- P5: decommission old sudo-bridge.

Opus+high prompt-craft upgrades over the Sonnet-era prompts (agent_prompts.md): bake in the 4-level test gate + agent_test_results logging (Sonnet's omitted it); split locked-contract from reference-implementation instead of embedding brittle literal scripts; right-size per-phase turn budgets; explicit "design is locked — do not redesign" block; per-phase rollback/abort criteria + idempotent pre-flight checks.