claude-projects/agent-builder/voice_chat_build_plan.md

# Voice Chat for Claude Code — Build Plan & Locked Decisions

# Preflight session: 2026-07-29 (claude-config conversation)
# HARD DEADLINE: functional by 2026-08-14 (Max subscription upgrade → up to 5 background agents run regularly)
# Status: DECISIONS LOCKED. Build not yet started.

---

## 0. One-paragraph summary

A fully-local, push-to-talk voice layer over Claude Code. You speak into a Bluetooth
headset on your laptop; a laptop-side daemon ships audio to server-01's GPU for
speech-to-text (Whisper), injects the transcript into Claude Code (running in tmux on
primary) via `tmux send-keys`; when Claude finishes, a Stop hook ships the response text
to server-01 for text-to-speech (XTTS-v2, Jarvis voice), and the audio is played back on
the headset. Every interactive session is a voice session. Up to 5 background agents each
run in their own tmux instance and speak through a single, non-overlapping playback queue;
you can say "pause <agent name>" to interrupt one, talk to it, then resume.

---

## 1. Topology (three machines)

| Machine | Role | Owns |
|---|---|---|
| **Laptop** (you) | Client you sit at; SSH into primary | WorkTunes Connect BT headset (mic + playback), the **audio daemon**, PTT hotkey, the **playback queue** |
| **Primary** (192.168.1.88) | Where Claude Code runs (you SSH in) | Claude Code in **tmux**, the **Stop hook**, `tmux send-keys` injection |
| **Server-01** (GPU) | Inference | **faster-whisper** (STT), **Coqui XTTS-v2** (TTS), Ollama (already there) |

Audio makes two network hops: laptop→server-01 (STT), server-01→laptop (playback).

---

## 2. LOCKED DECISIONS

### Interaction model — phased
- **Phase A (build now, due Aug 14):** dictation + read-back wrapper, single session, PTT, Jarvis voice, PLUS the full background-agent control layer (see §3). *The control layer is Phase A, not later — confirmed 2026-07-29.*
- **Phase B (later):** hands-free conversational loop (wake word / continuous listen), streaming-TTS polish.
- **Phase C (later):** custom character voices (e.g. Setsuna / Redo of Healer) + heavy speech-pattern / dictation styling layer.

### Engines — fully local, zero cloud, zero API cost
- **STT:** `faster-whisper` on server-01 GPU. (Aligns with `feedback_no_claude_api`, $20/mo cap, ambient-audio privacy.)
- **TTS:** **Coqui XTTS-v2** on server-01 GPU — voice-cloning capable, so Jarvis now and Setsuna later are just reference-sample swaps. (Piper rejected: fixed voices, can't clone.)

### Voice
- **Now:** Jarvis (Iron Man). Needs a 6–30s clean reference audio sample — **USER TO PROVIDE**.
- **Later (Phase C):** anime character voices + speech-pattern styling as a text-preprocessing layer before TTS.

### Activation
- **Push-to-talk** (hold key on laptop). Chosen for zero false triggers and because it mitigates the Bluetooth HFP/A2DP problem (mic only opens while held → playback stays high quality otherwise).

### Default behavior
- **Every interactive session is a voice session** (voice is the default / opt-OUT, not opt-in).
- Background/agent/non-voice contexts are gated separately (see §3, §4).

### Training data (`feedback_always_include_training_loop`)
- **Log the TEXT** of voice sessions (transcribed prompt + response) exactly as typed sessions are logged.
- **Never persist audio** — audio buffers deleted the moment STT/TTS completes. No conversation recordings.

---

## 3. Background-agent control layer (PHASE A)

This is the part that grew the scope and is required for the Aug-14 5-agent workflow.

1. **All agent outputs are spoken** — you want to hear every agent, not just the main session.
2. **Single arbitrated playback queue** on the laptop daemon: audio plays **consecutively, never overlapping**; each chunk is **source-tagged** (which agent / main session it came from) so you know who's talking.
3. **Voice control commands** are intercepted by the daemon as commands (NOT passed to Claude):
   - `pause <agent name>` → interrupt that agent's tmux instance so you can speak to/about it.
   - `resume <agent name>` → resume it.
   - (grammar extensible: `stop`, `skip`, `mute`, etc.)
4. **Per-agent tmux instances:** each background agent runs as its own Claude Code process in its own tmux window/pane. This is what makes independent pause/resume and clean per-source audio possible.

Open design question for build session: how "pause" is delivered to an agent's tmux pane
(Ctrl-C interrupt vs. a cooperative pause signal) without corrupting its run — resolve in
main session, do NOT let an agent guess.

---

## 4. Safety, blast radius, rollback

- **Blast radius concern:** hooks are GLOBAL (`/home/administrator/.claude/settings.json`) — the Stop hook fires for EVERY session and background agent. A naïve synchronous hook that waits on server-01 could freeze every conversation.
- **Containment:** the TTS Stop hook must be (a) gated so it only speaks in voice contexts, (b) **async / fire-and-forget with a short timeout**, (c) **fail-safe: any error → silent `exit 0`, never blocks Claude.**
- **Rollback:** the layer is purely additive. Disable = unset the voice env gate / remove the hook registration. tmux and Claude Code are unaffected. No Vault/N8N/Coolify schema changes.
- **Blast radius after containment:** voice sessions only; worst case = no audio, never a blocked Claude.

---

## 5. Hardware / latency notes

- **WorkTunes Connect (Bluetooth):** HFP/A2DP mutual exclusion — mic + high-quality playback can't coexist; PTT mitigates. Internal mic fine for commands, NOT good enough for recording a reference voice sample.
- **Headset upgrade (DEFERRED):** wired USB headset sidesteps HFP/A2DP entirely (clean mic + full-quality playback simultaneously). Recommend when ready; not a build blocker.
- **Latency budget (LAN + GPU):** ~1s before Claude thinks (BT capture + STT) + ~1–3s after (XTTS), dominated by Claude's own generation. Mitigate with **sentence-by-sentence streaming TTS** (Jarvis starts talking on sentence 1).

---

## 6. Deliverables of the preflight session

- [x] Locked grill-me decisions (this document).
- [ ] **WPM listening test** — script that synthesizes passages at 150/175/200/225/250 WPM in the Jarvis voice, asks a comprehension check after each, and reports your max accurate listening WPM → becomes the default TTS playback tempo. Doubles as first end-to-end server-01→laptop audio check. (Build + run once XTTS stands up.)
- [ ] **Build-plan agent prompts** (see §8) — bounded server-01 prompts + main-session checklist.

## 7. Open items / user tasks

- **Jarvis reference audio** (6–30s clean) — user provides.
- **server-01 GPU/VRAM audit** — can XTTS-v2 + Whisper + Ollama coexist in VRAM? (First build step / first agent task.)
- **Headset upgrade** — deferred decision.

---

## 8. Build execution shape (NOT one agent)

Per `playbook_background_agent_prompts.md` (agents = bounded ≤15 tool calls, defined
completion, no design) and the topology (agents run on primary, CANNOT touch the laptop),
the build splits into:

- **Bounded background-agent prompts (server-01 only):**
  - A1 — install + smoke-test `faster-whisper` on server-01 GPU; report VRAM headroom.
  - A2 — install + smoke-test XTTS-v2; clone from a provided Jarvis sample; emit a test WAV.
  - A3 — write + validate the fail-safe async Stop-hook TTS script (does NOT register it globally without confirmation).
- **Main-session work (you + Claude, cannot be agent-ized):**
  - Laptop audio daemon (mic, playback queue, PTT, control-command parser).
  - tmux injection wiring + per-agent tmux orchestration.
  - Global hook registration + gating + end-to-end integration test.

Agent prompts to be written into `voice_chat_agent_prompts.md` (alongside `agent_prompts.md`).