# Voice Chat for Claude Code — Build Plan & Locked Decisions # Preflight session: 2026-07-29 (claude-config conversation) # HARD DEADLINE: functional by 2026-08-14 (Max subscription upgrade → up to 5 background agents run regularly) # Status: DECISIONS LOCKED. Build not yet started. --- ## 0. One-paragraph summary A fully-local, push-to-talk voice layer over Claude Code. You speak into a Bluetooth headset on your laptop; a laptop-side daemon ships audio to server-01's GPU for speech-to-text (Whisper), injects the transcript into Claude Code (running in tmux on primary) via `tmux send-keys`; when Claude finishes, a Stop hook ships the response text to server-01 for text-to-speech (XTTS-v2, Jarvis voice), and the audio is played back on the headset. Every interactive session is a voice session. Up to 5 background agents each run in their own tmux instance and speak through a single, non-overlapping playback queue; you can say "pause " to interrupt one, talk to it, then resume. --- ## 1. Topology (three machines) | Machine | Role | Owns | |---|---|---| | **Laptop** (you) | Client you sit at; SSH into primary | WorkTunes Connect BT headset (mic + playback), the **audio daemon**, PTT hotkey, the **playback queue** | | **Primary** (192.168.1.88) | Where Claude Code runs (you SSH in) | Claude Code in **tmux**, the **Stop hook**, `tmux send-keys` injection | | **Server-01** (GPU) | Inference | **faster-whisper** (STT), **Coqui XTTS-v2** (TTS), Ollama (already there) | Audio makes two network hops: laptop→server-01 (STT), server-01→laptop (playback). --- ## 2. LOCKED DECISIONS ### Interaction model — phased - **Phase A (build now, due Aug 14):** dictation + read-back wrapper, single session, PTT, Jarvis voice, PLUS the full background-agent control layer (see §3). *The control layer is Phase A, not later — confirmed 2026-07-29.* - **Phase B (later):** hands-free conversational loop (wake word / continuous listen), streaming-TTS polish. - **Phase C (later):** custom character voices (e.g. Setsuna / Redo of Healer) + heavy speech-pattern / dictation styling layer. ### Engines — fully local, zero cloud, zero API cost - **STT:** `faster-whisper` on server-01 GPU. (Aligns with `feedback_no_claude_api`, $20/mo cap, ambient-audio privacy.) - **TTS:** **Coqui XTTS-v2** on server-01 GPU — voice-cloning capable, so Jarvis now and Setsuna later are just reference-sample swaps. (Piper rejected: fixed voices, can't clone.) ### Voice - **Now:** Jarvis (Iron Man). Needs a 6–30s clean reference audio sample — **USER TO PROVIDE**. - **Later (Phase C):** anime character voices + speech-pattern styling as a text-preprocessing layer before TTS. ### Activation - **Push-to-talk** (hold key on laptop). Chosen for zero false triggers and because it mitigates the Bluetooth HFP/A2DP problem (mic only opens while held → playback stays high quality otherwise). ### Default behavior - **Every interactive session is a voice session** (voice is the default / opt-OUT, not opt-in). - Background/agent/non-voice contexts are gated separately (see §3, §4). ### Training data (`feedback_always_include_training_loop`) - **Log the TEXT** of voice sessions (transcribed prompt + response) exactly as typed sessions are logged. - **Never persist audio** — audio buffers deleted the moment STT/TTS completes. No conversation recordings. --- ## 3. Background-agent control layer (PHASE A) This is the part that grew the scope and is required for the Aug-14 5-agent workflow. 1. **All agent outputs are spoken** — you want to hear every agent, not just the main session. 2. **Single arbitrated playback queue** on the laptop daemon: audio plays **consecutively, never overlapping**; each chunk is **source-tagged** (which agent / main session it came from) so you know who's talking. 3. **Voice control commands** are intercepted by the daemon as commands (NOT passed to Claude): - `pause ` → interrupt that agent's tmux instance so you can speak to/about it. - `resume ` → resume it. - (grammar extensible: `stop`, `skip`, `mute`, etc.) 4. **Per-agent tmux instances:** each background agent runs as its own Claude Code process in its own tmux window/pane. This is what makes independent pause/resume and clean per-source audio possible. Open design question for build session: how "pause" is delivered to an agent's tmux pane (Ctrl-C interrupt vs. a cooperative pause signal) without corrupting its run — resolve in main session, do NOT let an agent guess. --- ## 4. Safety, blast radius, rollback - **Blast radius concern:** hooks are GLOBAL (`/home/administrator/.claude/settings.json`) — the Stop hook fires for EVERY session and background agent. A naïve synchronous hook that waits on server-01 could freeze every conversation. - **Containment:** the TTS Stop hook must be (a) gated so it only speaks in voice contexts, (b) **async / fire-and-forget with a short timeout**, (c) **fail-safe: any error → silent `exit 0`, never blocks Claude.** - **Rollback:** the layer is purely additive. Disable = unset the voice env gate / remove the hook registration. tmux and Claude Code are unaffected. No Vault/N8N/Coolify schema changes. - **Blast radius after containment:** voice sessions only; worst case = no audio, never a blocked Claude. --- ## 5. Hardware / latency notes - **WorkTunes Connect (Bluetooth):** HFP/A2DP mutual exclusion — mic + high-quality playback can't coexist; PTT mitigates. Internal mic fine for commands, NOT good enough for recording a reference voice sample. - **Headset upgrade (DEFERRED):** wired USB headset sidesteps HFP/A2DP entirely (clean mic + full-quality playback simultaneously). Recommend when ready; not a build blocker. - **Latency budget (LAN + GPU):** ~1s before Claude thinks (BT capture + STT) + ~1–3s after (XTTS), dominated by Claude's own generation. Mitigate with **sentence-by-sentence streaming TTS** (Jarvis starts talking on sentence 1). --- ## 6. Deliverables of the preflight session - [x] Locked grill-me decisions (this document). - [ ] **WPM listening test** — script that synthesizes passages at 150/175/200/225/250 WPM in the Jarvis voice, asks a comprehension check after each, and reports your max accurate listening WPM → becomes the default TTS playback tempo. Doubles as first end-to-end server-01→laptop audio check. (Build + run once XTTS stands up.) - [ ] **Build-plan agent prompts** (see §8) — bounded server-01 prompts + main-session checklist. ## 7. Open items / user tasks - **Jarvis reference audio** (6–30s clean) — user provides. - **server-01 GPU/VRAM audit** — can XTTS-v2 + Whisper + Ollama coexist in VRAM? (First build step / first agent task.) - **Headset upgrade** — deferred decision. --- ## 8. Build execution shape (NOT one agent) Per `playbook_background_agent_prompts.md` (agents = bounded ≤15 tool calls, defined completion, no design) and the topology (agents run on primary, CANNOT touch the laptop), the build splits into: - **Bounded background-agent prompts (server-01 only):** - A1 — install + smoke-test `faster-whisper` on server-01 GPU; report VRAM headroom. - A2 — install + smoke-test XTTS-v2; clone from a provided Jarvis sample; emit a test WAV. - A3 — write + validate the fail-safe async Stop-hook TTS script (does NOT register it globally without confirmation). - **Main-session work (you + Claude, cannot be agent-ized):** - Laptop audio daemon (mic, playback queue, PTT, control-command parser). - tmux injection wiring + per-agent tmux orchestration. - Global hook registration + gating + end-to-end integration test. Agent prompts to be written into `voice_chat_agent_prompts.md` (alongside `agent_prompts.md`).