docs(voice-chat): lock preflight decisions + decomposed build prompts
Voice Chat for Claude Code preflight/design session (personal_projects id=214). Fully-local PTT voice: faster-whisper + Coqui XTTS-v2 on server-01 GPU, Jarvis voice. Phase A includes background-agent audio-arbitration queue + pause/resume voice control + per-agent tmux. Hard deadline ~2026-08-14 (Max upgrade). - voice_chat_build_plan.md: locked decisions, topology, blast-radius/rollback - voice_chat_agent_prompts.md: 3 bounded server-01 agent prompts (A1 Whisper, A2 XTTS+Jarvis, A3 fail-safe Stop-hook) + main-session checklist - context.md: refresh CURRENT STATE block (was stale since 2026-06-04) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Backtalk6858
@@ -0,0 +1,221 @@
|
||||
# Voice Chat — Background Agent Prompts (decomposed build)
|
||||
|
||||
# Companion to voice_chat_build_plan.md. Preflight: 2026-07-29.
|
||||
# Launch target: on/before the build session ahead of 2026-08-14.
|
||||
#
|
||||
# ⚠️ RE-VERIFY AT LAUNCH (infra facts decay — do NOT trust these blindly):
|
||||
# - server-01 access = sudo-bridge-server01 at http://192.168.1.90:8082 (NOT secrets-proxy /shell).
|
||||
# Confirm the bridge is up and the endpoint/port before launching.
|
||||
# - Vault: resolve IP dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40; AppRole only.
|
||||
# - infrastructure-auth-map: secret/data/claude/infrastructure-auth-map.
|
||||
# - server-01 file writes: use `docker exec <container> python3 -c` (tier-1) for unattended writes;
|
||||
# bare `python3 -c *` may be tier-2 (NTFY approval). Confirm allowlist state first.
|
||||
# - These 3 agents touch server-01 GPU/models independently → safe to run in PARALLEL,
|
||||
# EXCEPT A1 and A2 both measure VRAM; run A1 first, then A2, so VRAM numbers are additive/coherent.
|
||||
# - A3 runs on PRIMARY (writes a hook script), independent of A1/A2 → fully parallel.
|
||||
#
|
||||
# NONE of these agents register the global hook, install the laptop daemon, or make design
|
||||
# decisions. Those are main-session (see §MAIN-SESSION CHECKLIST at the bottom).
|
||||
|
||||
---
|
||||
|
||||
## AGENT A1 — Install + smoke-test faster-whisper on server-01
|
||||
|
||||
```
|
||||
You are a setup agent operating on server-01 through the sudo-bridge-server01 endpoint
|
||||
(http://192.168.1.90:8082 — VERIFY it is reachable before doing anything else). You will
|
||||
install and smoke-test faster-whisper for GPU speech-to-text. You will NOT touch primary,
|
||||
the laptop, Ollama's config, or any global hook.
|
||||
|
||||
## Task
|
||||
Get faster-whisper transcribing on server-01's GPU and report exact VRAM headroom while
|
||||
Ollama is running, so we know STT can coexist with Ollama + (later) XTTS.
|
||||
|
||||
## Credentials you will need
|
||||
- server-01 privileged ops: sudo-bridge-server01 at http://192.168.1.90:8082 — auth key at
|
||||
Vault secret/data/claude/infrastructure-auth-map (field: sudo-bridge-server01). RE-VERIFY field name.
|
||||
- AppRole bootstrap (if Vault read needed):
|
||||
VAULT_IP=$(docker inspect vault-iwaulpoi5hwirdlogshmul40 --format '{{json .NetworkSettings.Networks}}' | python3 -c "import sys,json; nets=json.load(sys.stdin); print(list(nets.values())[0]['IPAddress'])")
|
||||
ROLE_ID=$(cat /opt/appdata/docker/docker-compose/vault/approle/role-id | tr -d '[:space:]')
|
||||
SECRET_ID=$(cat /opt/appdata/docker/docker-compose/vault/approle/secret-id | tr -d '[:space:]')
|
||||
# login → token; ALWAYS revoke-self after use.
|
||||
|
||||
## Step-by-step (DISCOVERY FIRST — do not assume the environment)
|
||||
1. Discover: GPU model, driver + CUDA version (`nvidia-smi`), total VRAM, and VRAM currently
|
||||
used by Ollama with a model loaded. Discover whether server-01 runs models bare-metal
|
||||
(python venv) or in Docker, and where Ollama lives. Record all of this.
|
||||
2. Choose model size from measured free VRAM: prefer `large-v3` or `distil-large-v3` if it
|
||||
fits alongside Ollama with margin; else fall back to `medium`. State the choice + why.
|
||||
3. Install faster-whisper in the environment matching step 1 (venv or a container). Prefer the
|
||||
least-invasive path; do NOT disturb Ollama.
|
||||
4. Smoke test: transcribe a short built-in/test audio clip (generate one with `say`/espeak or
|
||||
a tiny recorded WAV) and confirm the returned text is correct.
|
||||
5. Report the exact invocation (command or endpoint) that main-session code will call, plus
|
||||
measured VRAM free with Ollama + Whisper both resident.
|
||||
|
||||
## Output (in addition to the mandatory wrap-up)
|
||||
Report: GPU/CUDA, environment type (venv/docker + path), chosen Whisper model + rationale,
|
||||
the exact transcribe invocation, and VRAM: total / Ollama-resident / Whisper-resident / free.
|
||||
|
||||
## MANDATORY WRAP-UP (required regardless of success or failure)
|
||||
|
||||
Before stopping for ANY reason — task complete, error, or approaching turn limit — output this JSON as your final message. Do not stop without it.
|
||||
|
||||
{
|
||||
"status": "succeeded|partially_succeeded|failed",
|
||||
"actions_taken": ["action 1 — outcome", "action 2 — outcome"],
|
||||
"actions_failed": ["action — reason it failed"],
|
||||
"notes": "anything relevant for the next session"
|
||||
}
|
||||
|
||||
If you hit --max-turns before finishing, set status="partially_succeeded" and list what remains in actions_failed.
|
||||
|
||||
--max-turns 15
|
||||
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## AGENT A2 — Install XTTS-v2 + clone the Jarvis voice on server-01
|
||||
|
||||
```
|
||||
You are a setup agent operating on server-01 through sudo-bridge-server01
|
||||
(http://192.168.1.90:8082 — VERIFY reachable first). You will install Coqui XTTS-v2 and
|
||||
produce a test WAV in a cloned voice. You will NOT touch primary, the laptop, or any hook.
|
||||
RUN AFTER A1 so VRAM numbers are coherent.
|
||||
|
||||
## Task
|
||||
Stand up XTTS-v2 for GPU voice-cloning TTS, clone from the provided reference sample, and
|
||||
emit a test WAV. Measure real-time factor (RTF) and VRAM with Ollama (+ Whisper if resident).
|
||||
|
||||
## Context / pre-fetched data
|
||||
- Reference voice sample (Jarvis), 6–30s clean mono WAV, will be at:
|
||||
<<JARVIS_REFERENCE_WAV_PATH — USER PROVIDES; substitute the real path at launch>>
|
||||
- Engine: Coqui XTTS-v2 (model `tts_models/multilingual/multi-dataset/xtts_v2`).
|
||||
NOTE its license is non-commercial (Coqui Public Model License) — fine for personal use.
|
||||
|
||||
## Credentials you will need
|
||||
- sudo-bridge-server01 auth: Vault secret/data/claude/infrastructure-auth-map
|
||||
(field: sudo-bridge-server01). RE-VERIFY.
|
||||
- AppRole bootstrap as in A1 if a Vault read is needed; revoke-self after.
|
||||
|
||||
## Step-by-step (DISCOVERY FIRST)
|
||||
1. Confirm the environment discovered in A1 (venv vs docker, CUDA). Reuse it; do not rebuild.
|
||||
2. Install Coqui XTTS-v2 (the maintained `coqui-tts` fork) in that environment. Accept the
|
||||
model license non-interactively (set the env var the library expects).
|
||||
3. Confirm the reference WAV exists at the provided path and is mono, 16k+ Hz. If missing,
|
||||
STOP and report — do not fabricate a voice.
|
||||
4. Synthesize one test sentence (e.g. "All systems are online, sir.") cloning the reference,
|
||||
write it to a test WAV, and report the path.
|
||||
5. Measure RTF (synthesis time / audio duration) and VRAM: total / free with Ollama
|
||||
(+ Whisper if A1's model is still resident) / XTTS-resident.
|
||||
6. Report the exact synthesis invocation main-session code will call, and whether XTTS +
|
||||
Whisper + Ollama all fit in VRAM simultaneously (the key coexistence question).
|
||||
|
||||
## Output (in addition to the mandatory wrap-up)
|
||||
Report: install path, test WAV path, RTF, VRAM breakdown, coexistence verdict (all three fit? y/n),
|
||||
and the exact synthesis invocation.
|
||||
|
||||
## MANDATORY WRAP-UP (required regardless of success or failure)
|
||||
|
||||
Before stopping for ANY reason — task complete, error, or approaching turn limit — output this JSON as your final message. Do not stop without it.
|
||||
|
||||
{
|
||||
"status": "succeeded|partially_succeeded|failed",
|
||||
"actions_taken": ["action 1 — outcome", "action 2 — outcome"],
|
||||
"actions_failed": ["action — reason it failed"],
|
||||
"notes": "anything relevant for the next session"
|
||||
}
|
||||
|
||||
If you hit --max-turns before finishing, set status="partially_succeeded" and list what remains in actions_failed.
|
||||
|
||||
--max-turns 15
|
||||
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## AGENT A3 — Write + validate the fail-safe Stop-hook TTS script (primary)
|
||||
|
||||
```
|
||||
You are a script-authoring agent on the primary server (192.168.1.88). You will WRITE and
|
||||
VALIDATE a fail-safe Stop-hook script that sends Claude's response text to server-01's XTTS
|
||||
endpoint. You will NOT register it in settings.json (that is a human main-session step) and
|
||||
you will NOT touch server-01 or the laptop.
|
||||
|
||||
## Task
|
||||
Create /opt/appdata/docker/.claude/hooks/voice_tts_stop.py implementing a Stop hook that:
|
||||
1. Reads the hook JSON from stdin.
|
||||
2. GATE: if env var CLAUDE_VOICE != "1", exit 0 immediately (silent no-op). This is what
|
||||
keeps every non-voice session and background-agent context silent.
|
||||
3. Extracts Claude's final response text from the hook payload.
|
||||
4. Fires the text to the server-01 XTTS endpoint as FIRE-AND-FORGET with a SHORT timeout
|
||||
(<= 1.5s connect). Do not wait for audio; the laptop daemon handles playback + queueing.
|
||||
5. FAIL-SAFE: wrap everything so ANY exception, timeout, or bad payload → exit 0 silently.
|
||||
The hook must NEVER raise, never block, never non-zero. Blast radius = zero.
|
||||
6. Tag the request with the session/source id so the laptop playback queue can label it.
|
||||
|
||||
## Context / pre-fetched data
|
||||
- The XTTS endpoint URL is not final yet (A2 produces it). Read it from an env var
|
||||
VOICE_TTS_URL (default to a placeholder) so main-session can wire the real value without
|
||||
editing the script.
|
||||
- Hook JSON shape: confirm the current Stop-hook stdin schema against an existing hook in
|
||||
/opt/appdata/docker/.claude/hooks/ before parsing (do NOT guess field names).
|
||||
|
||||
## Step-by-step
|
||||
1. Read one existing hook in /opt/appdata/docker/.claude/hooks/ to confirm stdin JSON shape
|
||||
and the house style.
|
||||
2. Write voice_tts_stop.py per the Task spec above. Use only stdlib (urllib, json, os).
|
||||
3. Validate syntax: python3 -c "import ast; ast.parse(open('/opt/appdata/docker/.claude/hooks/voice_tts_stop.py').read())"
|
||||
4. Test the gate: run the script with CLAUDE_VOICE unset piping a sample payload → must exit 0
|
||||
with no network call. Then with CLAUDE_VOICE=1 and VOICE_TTS_URL pointed at a dead port →
|
||||
must STILL exit 0 (proving fail-safe), just no audio.
|
||||
5. Do NOT edit settings.json. Report the exact settings.json Stop-hook registration snippet
|
||||
for the human to paste later.
|
||||
|
||||
## Output (in addition to the mandatory wrap-up)
|
||||
Report: script path, ast.parse result, both gate-test outcomes, and the settings.json snippet.
|
||||
|
||||
## MANDATORY WRAP-UP (required regardless of success or failure)
|
||||
|
||||
Before stopping for ANY reason — task complete, error, or approaching turn limit — output this JSON as your final message. Do not stop without it.
|
||||
|
||||
{
|
||||
"status": "succeeded|partially_succeeded|failed",
|
||||
"actions_taken": ["action 1 — outcome", "action 2 — outcome"],
|
||||
"actions_failed": ["action — reason it failed"],
|
||||
"notes": "anything relevant for the next session"
|
||||
}
|
||||
|
||||
If you hit --max-turns before finishing, set status="partially_succeeded" and list what remains in actions_failed.
|
||||
|
||||
--max-turns 15
|
||||
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MAIN-SESSION CHECKLIST (NOT agent-izable — you + Claude in the build session)
|
||||
|
||||
These cannot be background agents: they need design decisions and/or live on the LAPTOP,
|
||||
which agents (running on primary) cannot reach.
|
||||
|
||||
1. **Laptop audio daemon** — one local service that owns:
|
||||
- Mic capture from the WorkTunes headset + PTT hotkey (hold-to-talk).
|
||||
- STT client: POST captured audio to server-01 Whisper (A1's invocation) → get text.
|
||||
- Injection: SSH to primary and `tmux send-keys` the text into the target Claude pane.
|
||||
- Playback queue: single FIFO, consecutive/non-overlapping, source-tagged (main + each agent).
|
||||
- Control-command parser: intercept `pause <agent>` / `resume <agent>` (and future verbs)
|
||||
as commands, act on the agent's tmux instance instead of sending to Claude.
|
||||
2. **tmux orchestration** — main Claude pane + one pane/window per background agent (up to 5),
|
||||
named so the control parser can target them. Decide the pause mechanism (Ctrl-C interrupt
|
||||
vs. cooperative signal) here — do NOT let an agent guess it.
|
||||
3. **Global hook registration** — paste A3's Stop-hook snippet into settings.json; set the
|
||||
voice env gate in the interactive launch wrapper (so voice = default for interactive,
|
||||
silent for background/agent contexts). End-to-end test the gate.
|
||||
4. **Streaming TTS** — wire sentence-by-sentence synthesis so Jarvis starts talking on sentence 1.
|
||||
5. **WPM listening test** — build + run once XTTS is up (A2): synthesize passages at
|
||||
150/175/200/225/250 WPM in the Jarvis voice, comprehension check after each, record max
|
||||
accurate WPM → set as default playback tempo.
|
||||
6. **Training-data logging** — voice-session TEXT logged like typed sessions; audio never persisted.
|
||||
```
|
||||
@@ -0,0 +1,126 @@
|
||||
# Voice Chat for Claude Code — Build Plan & Locked Decisions
|
||||
|
||||
# Preflight session: 2026-07-29 (claude-config conversation)
|
||||
# HARD DEADLINE: functional by 2026-08-14 (Max subscription upgrade → up to 5 background agents run regularly)
|
||||
# Status: DECISIONS LOCKED. Build not yet started.
|
||||
|
||||
---
|
||||
|
||||
## 0. One-paragraph summary
|
||||
|
||||
A fully-local, push-to-talk voice layer over Claude Code. You speak into a Bluetooth
|
||||
headset on your laptop; a laptop-side daemon ships audio to server-01's GPU for
|
||||
speech-to-text (Whisper), injects the transcript into Claude Code (running in tmux on
|
||||
primary) via `tmux send-keys`; when Claude finishes, a Stop hook ships the response text
|
||||
to server-01 for text-to-speech (XTTS-v2, Jarvis voice), and the audio is played back on
|
||||
the headset. Every interactive session is a voice session. Up to 5 background agents each
|
||||
run in their own tmux instance and speak through a single, non-overlapping playback queue;
|
||||
you can say "pause <agent name>" to interrupt one, talk to it, then resume.
|
||||
|
||||
---
|
||||
|
||||
## 1. Topology (three machines)
|
||||
|
||||
| Machine | Role | Owns |
|
||||
|---|---|---|
|
||||
| **Laptop** (you) | Client you sit at; SSH into primary | WorkTunes Connect BT headset (mic + playback), the **audio daemon**, PTT hotkey, the **playback queue** |
|
||||
| **Primary** (192.168.1.88) | Where Claude Code runs (you SSH in) | Claude Code in **tmux**, the **Stop hook**, `tmux send-keys` injection |
|
||||
| **Server-01** (GPU) | Inference | **faster-whisper** (STT), **Coqui XTTS-v2** (TTS), Ollama (already there) |
|
||||
|
||||
Audio makes two network hops: laptop→server-01 (STT), server-01→laptop (playback).
|
||||
|
||||
---
|
||||
|
||||
## 2. LOCKED DECISIONS
|
||||
|
||||
### Interaction model — phased
|
||||
- **Phase A (build now, due Aug 14):** dictation + read-back wrapper, single session, PTT, Jarvis voice, PLUS the full background-agent control layer (see §3). *The control layer is Phase A, not later — confirmed 2026-07-29.*
|
||||
- **Phase B (later):** hands-free conversational loop (wake word / continuous listen), streaming-TTS polish.
|
||||
- **Phase C (later):** custom character voices (e.g. Setsuna / Redo of Healer) + heavy speech-pattern / dictation styling layer.
|
||||
|
||||
### Engines — fully local, zero cloud, zero API cost
|
||||
- **STT:** `faster-whisper` on server-01 GPU. (Aligns with `feedback_no_claude_api`, $20/mo cap, ambient-audio privacy.)
|
||||
- **TTS:** **Coqui XTTS-v2** on server-01 GPU — voice-cloning capable, so Jarvis now and Setsuna later are just reference-sample swaps. (Piper rejected: fixed voices, can't clone.)
|
||||
|
||||
### Voice
|
||||
- **Now:** Jarvis (Iron Man). Needs a 6–30s clean reference audio sample — **USER TO PROVIDE**.
|
||||
- **Later (Phase C):** anime character voices + speech-pattern styling as a text-preprocessing layer before TTS.
|
||||
|
||||
### Activation
|
||||
- **Push-to-talk** (hold key on laptop). Chosen for zero false triggers and because it mitigates the Bluetooth HFP/A2DP problem (mic only opens while held → playback stays high quality otherwise).
|
||||
|
||||
### Default behavior
|
||||
- **Every interactive session is a voice session** (voice is the default / opt-OUT, not opt-in).
|
||||
- Background/agent/non-voice contexts are gated separately (see §3, §4).
|
||||
|
||||
### Training data (`feedback_always_include_training_loop`)
|
||||
- **Log the TEXT** of voice sessions (transcribed prompt + response) exactly as typed sessions are logged.
|
||||
- **Never persist audio** — audio buffers deleted the moment STT/TTS completes. No conversation recordings.
|
||||
|
||||
---
|
||||
|
||||
## 3. Background-agent control layer (PHASE A)
|
||||
|
||||
This is the part that grew the scope and is required for the Aug-14 5-agent workflow.
|
||||
|
||||
1. **All agent outputs are spoken** — you want to hear every agent, not just the main session.
|
||||
2. **Single arbitrated playback queue** on the laptop daemon: audio plays **consecutively, never overlapping**; each chunk is **source-tagged** (which agent / main session it came from) so you know who's talking.
|
||||
3. **Voice control commands** are intercepted by the daemon as commands (NOT passed to Claude):
|
||||
- `pause <agent name>` → interrupt that agent's tmux instance so you can speak to/about it.
|
||||
- `resume <agent name>` → resume it.
|
||||
- (grammar extensible: `stop`, `skip`, `mute`, etc.)
|
||||
4. **Per-agent tmux instances:** each background agent runs as its own Claude Code process in its own tmux window/pane. This is what makes independent pause/resume and clean per-source audio possible.
|
||||
|
||||
Open design question for build session: how "pause" is delivered to an agent's tmux pane
|
||||
(Ctrl-C interrupt vs. a cooperative pause signal) without corrupting its run — resolve in
|
||||
main session, do NOT let an agent guess.
|
||||
|
||||
---
|
||||
|
||||
## 4. Safety, blast radius, rollback
|
||||
|
||||
- **Blast radius concern:** hooks are GLOBAL (`/home/administrator/.claude/settings.json`) — the Stop hook fires for EVERY session and background agent. A naïve synchronous hook that waits on server-01 could freeze every conversation.
|
||||
- **Containment:** the TTS Stop hook must be (a) gated so it only speaks in voice contexts, (b) **async / fire-and-forget with a short timeout**, (c) **fail-safe: any error → silent `exit 0`, never blocks Claude.**
|
||||
- **Rollback:** the layer is purely additive. Disable = unset the voice env gate / remove the hook registration. tmux and Claude Code are unaffected. No Vault/N8N/Coolify schema changes.
|
||||
- **Blast radius after containment:** voice sessions only; worst case = no audio, never a blocked Claude.
|
||||
|
||||
---
|
||||
|
||||
## 5. Hardware / latency notes
|
||||
|
||||
- **WorkTunes Connect (Bluetooth):** HFP/A2DP mutual exclusion — mic + high-quality playback can't coexist; PTT mitigates. Internal mic fine for commands, NOT good enough for recording a reference voice sample.
|
||||
- **Headset upgrade (DEFERRED):** wired USB headset sidesteps HFP/A2DP entirely (clean mic + full-quality playback simultaneously). Recommend when ready; not a build blocker.
|
||||
- **Latency budget (LAN + GPU):** ~1s before Claude thinks (BT capture + STT) + ~1–3s after (XTTS), dominated by Claude's own generation. Mitigate with **sentence-by-sentence streaming TTS** (Jarvis starts talking on sentence 1).
|
||||
|
||||
---
|
||||
|
||||
## 6. Deliverables of the preflight session
|
||||
|
||||
- [x] Locked grill-me decisions (this document).
|
||||
- [ ] **WPM listening test** — script that synthesizes passages at 150/175/200/225/250 WPM in the Jarvis voice, asks a comprehension check after each, and reports your max accurate listening WPM → becomes the default TTS playback tempo. Doubles as first end-to-end server-01→laptop audio check. (Build + run once XTTS stands up.)
|
||||
- [ ] **Build-plan agent prompts** (see §8) — bounded server-01 prompts + main-session checklist.
|
||||
|
||||
## 7. Open items / user tasks
|
||||
|
||||
- **Jarvis reference audio** (6–30s clean) — user provides.
|
||||
- **server-01 GPU/VRAM audit** — can XTTS-v2 + Whisper + Ollama coexist in VRAM? (First build step / first agent task.)
|
||||
- **Headset upgrade** — deferred decision.
|
||||
|
||||
---
|
||||
|
||||
## 8. Build execution shape (NOT one agent)
|
||||
|
||||
Per `playbook_background_agent_prompts.md` (agents = bounded ≤15 tool calls, defined
|
||||
completion, no design) and the topology (agents run on primary, CANNOT touch the laptop),
|
||||
the build splits into:
|
||||
|
||||
- **Bounded background-agent prompts (server-01 only):**
|
||||
- A1 — install + smoke-test `faster-whisper` on server-01 GPU; report VRAM headroom.
|
||||
- A2 — install + smoke-test XTTS-v2; clone from a provided Jarvis sample; emit a test WAV.
|
||||
- A3 — write + validate the fail-safe async Stop-hook TTS script (does NOT register it globally without confirmation).
|
||||
- **Main-session work (you + Claude, cannot be agent-ized):**
|
||||
- Laptop audio daemon (mic, playback queue, PTT, control-command parser).
|
||||
- tmux injection wiring + per-agent tmux orchestration.
|
||||
- Global hook registration + gating + end-to-end integration test.
|
||||
|
||||
Agent prompts to be written into `voice_chat_agent_prompts.md` (alongside `agent_prompts.md`).
|
||||
Reference in New Issue
Block a user