diff --git a/.claude/context.md b/.claude/context.md index 64c65ac..a8f4376 100644 --- a/.claude/context.md +++ b/.claude/context.md @@ -37,7 +37,7 @@ Saturday/Sunday Claude configuration and improvement sessions — behavior_chang - pgvector extension in claude_config DB — required for claude_memories table ## Current state -2026-06-04 Thursday: git criteria universal playbook built (personal_projects id=38 → completed). playbook_git_criteria_universal.md created in both memory dirs — single authorized trigger (checklist only), partial commit on hard block, universal pre-stage registry, dynamic co-author line, two-layer evolution. playbook_git_commits.md updated to extend it. media-api.py + related files still uncommitted — containerized script, testing not confirmed. Next session: security vulnerability patching (~6:30 PM Thursday). +2026-07-29 Wednesday: **Voice Chat for Claude Code — preflight/design session** (personal_projects id=214, in_progress). Ran full grill-me; locked all decisions. DELIVERABLES (in agent-builder/, committed as files): `voice_chat_build_plan.md` (locked decisions) + `voice_chat_agent_prompts.md` (3 bounded server-01 agent prompts A1 Whisper / A2 XTTS+Jarvis / A3 Stop-hook + main-session checklist). New memory: `project_voice_chat_claude_code.md` (indexed). Design: fully-local PTT voice (faster-whisper + Coqui XTTS-v2 on server-01 GPU, Jarvis voice), laptop audio daemon ⇄ primary Claude-in-tmux ⇄ server-01 GPU. Phase A includes background-agent audio-arbitration queue + `pause/resume ` voice control + per-agent tmux instances. HARD DEADLINE ~2026-08-14 (Max upgrade). Open user tasks: provide Jarvis reference WAV; A1/A2 answer VRAM coexistence; build WPM listening test. NEXT: switch to infrastructure conversation for Tailscale/Twingate preflight (relevant to voice: transport for laptop↔server audio hops if laptop ever remote), then media_pipeline (personal_projects id=12). ## Sunday Dev Plan (13 items) **Hooks:** diff --git a/agent-builder/voice_chat_agent_prompts.md b/agent-builder/voice_chat_agent_prompts.md new file mode 100644 index 0000000..ffb77c9 --- /dev/null +++ b/agent-builder/voice_chat_agent_prompts.md @@ -0,0 +1,221 @@ +# Voice Chat — Background Agent Prompts (decomposed build) + +# Companion to voice_chat_build_plan.md. Preflight: 2026-07-29. +# Launch target: on/before the build session ahead of 2026-08-14. +# +# ⚠️ RE-VERIFY AT LAUNCH (infra facts decay — do NOT trust these blindly): +# - server-01 access = sudo-bridge-server01 at http://192.168.1.90:8082 (NOT secrets-proxy /shell). +# Confirm the bridge is up and the endpoint/port before launching. +# - Vault: resolve IP dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40; AppRole only. +# - infrastructure-auth-map: secret/data/claude/infrastructure-auth-map. +# - server-01 file writes: use `docker exec python3 -c` (tier-1) for unattended writes; +# bare `python3 -c *` may be tier-2 (NTFY approval). Confirm allowlist state first. +# - These 3 agents touch server-01 GPU/models independently → safe to run in PARALLEL, +# EXCEPT A1 and A2 both measure VRAM; run A1 first, then A2, so VRAM numbers are additive/coherent. +# - A3 runs on PRIMARY (writes a hook script), independent of A1/A2 → fully parallel. +# +# NONE of these agents register the global hook, install the laptop daemon, or make design +# decisions. Those are main-session (see §MAIN-SESSION CHECKLIST at the bottom). + +--- + +## AGENT A1 — Install + smoke-test faster-whisper on server-01 + +``` +You are a setup agent operating on server-01 through the sudo-bridge-server01 endpoint +(http://192.168.1.90:8082 — VERIFY it is reachable before doing anything else). You will +install and smoke-test faster-whisper for GPU speech-to-text. You will NOT touch primary, +the laptop, Ollama's config, or any global hook. + +## Task +Get faster-whisper transcribing on server-01's GPU and report exact VRAM headroom while +Ollama is running, so we know STT can coexist with Ollama + (later) XTTS. + +## Credentials you will need +- server-01 privileged ops: sudo-bridge-server01 at http://192.168.1.90:8082 — auth key at + Vault secret/data/claude/infrastructure-auth-map (field: sudo-bridge-server01). RE-VERIFY field name. +- AppRole bootstrap (if Vault read needed): + VAULT_IP=$(docker inspect vault-iwaulpoi5hwirdlogshmul40 --format '{{json .NetworkSettings.Networks}}' | python3 -c "import sys,json; nets=json.load(sys.stdin); print(list(nets.values())[0]['IPAddress'])") + ROLE_ID=$(cat /opt/appdata/docker/docker-compose/vault/approle/role-id | tr -d '[:space:]') + SECRET_ID=$(cat /opt/appdata/docker/docker-compose/vault/approle/secret-id | tr -d '[:space:]') + # login → token; ALWAYS revoke-self after use. + +## Step-by-step (DISCOVERY FIRST — do not assume the environment) +1. Discover: GPU model, driver + CUDA version (`nvidia-smi`), total VRAM, and VRAM currently + used by Ollama with a model loaded. Discover whether server-01 runs models bare-metal + (python venv) or in Docker, and where Ollama lives. Record all of this. +2. Choose model size from measured free VRAM: prefer `large-v3` or `distil-large-v3` if it + fits alongside Ollama with margin; else fall back to `medium`. State the choice + why. +3. Install faster-whisper in the environment matching step 1 (venv or a container). Prefer the + least-invasive path; do NOT disturb Ollama. +4. Smoke test: transcribe a short built-in/test audio clip (generate one with `say`/espeak or + a tiny recorded WAV) and confirm the returned text is correct. +5. Report the exact invocation (command or endpoint) that main-session code will call, plus + measured VRAM free with Ollama + Whisper both resident. + +## Output (in addition to the mandatory wrap-up) +Report: GPU/CUDA, environment type (venv/docker + path), chosen Whisper model + rationale, +the exact transcribe invocation, and VRAM: total / Ollama-resident / Whisper-resident / free. + +## MANDATORY WRAP-UP (required regardless of success or failure) + +Before stopping for ANY reason — task complete, error, or approaching turn limit — output this JSON as your final message. Do not stop without it. + +{ + "status": "succeeded|partially_succeeded|failed", + "actions_taken": ["action 1 — outcome", "action 2 — outcome"], + "actions_failed": ["action — reason it failed"], + "notes": "anything relevant for the next session" +} + +If you hit --max-turns before finishing, set status="partially_succeeded" and list what remains in actions_failed. + +--max-turns 15 +If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded. +``` + +--- + +## AGENT A2 — Install XTTS-v2 + clone the Jarvis voice on server-01 + +``` +You are a setup agent operating on server-01 through sudo-bridge-server01 +(http://192.168.1.90:8082 — VERIFY reachable first). You will install Coqui XTTS-v2 and +produce a test WAV in a cloned voice. You will NOT touch primary, the laptop, or any hook. +RUN AFTER A1 so VRAM numbers are coherent. + +## Task +Stand up XTTS-v2 for GPU voice-cloning TTS, clone from the provided reference sample, and +emit a test WAV. Measure real-time factor (RTF) and VRAM with Ollama (+ Whisper if resident). + +## Context / pre-fetched data +- Reference voice sample (Jarvis), 6–30s clean mono WAV, will be at: + <> +- Engine: Coqui XTTS-v2 (model `tts_models/multilingual/multi-dataset/xtts_v2`). + NOTE its license is non-commercial (Coqui Public Model License) — fine for personal use. + +## Credentials you will need +- sudo-bridge-server01 auth: Vault secret/data/claude/infrastructure-auth-map + (field: sudo-bridge-server01). RE-VERIFY. +- AppRole bootstrap as in A1 if a Vault read is needed; revoke-self after. + +## Step-by-step (DISCOVERY FIRST) +1. Confirm the environment discovered in A1 (venv vs docker, CUDA). Reuse it; do not rebuild. +2. Install Coqui XTTS-v2 (the maintained `coqui-tts` fork) in that environment. Accept the + model license non-interactively (set the env var the library expects). +3. Confirm the reference WAV exists at the provided path and is mono, 16k+ Hz. If missing, + STOP and report — do not fabricate a voice. +4. Synthesize one test sentence (e.g. "All systems are online, sir.") cloning the reference, + write it to a test WAV, and report the path. +5. Measure RTF (synthesis time / audio duration) and VRAM: total / free with Ollama + (+ Whisper if A1's model is still resident) / XTTS-resident. +6. Report the exact synthesis invocation main-session code will call, and whether XTTS + + Whisper + Ollama all fit in VRAM simultaneously (the key coexistence question). + +## Output (in addition to the mandatory wrap-up) +Report: install path, test WAV path, RTF, VRAM breakdown, coexistence verdict (all three fit? y/n), +and the exact synthesis invocation. + +## MANDATORY WRAP-UP (required regardless of success or failure) + +Before stopping for ANY reason — task complete, error, or approaching turn limit — output this JSON as your final message. Do not stop without it. + +{ + "status": "succeeded|partially_succeeded|failed", + "actions_taken": ["action 1 — outcome", "action 2 — outcome"], + "actions_failed": ["action — reason it failed"], + "notes": "anything relevant for the next session" +} + +If you hit --max-turns before finishing, set status="partially_succeeded" and list what remains in actions_failed. + +--max-turns 15 +If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded. +``` + +--- + +## AGENT A3 — Write + validate the fail-safe Stop-hook TTS script (primary) + +``` +You are a script-authoring agent on the primary server (192.168.1.88). You will WRITE and +VALIDATE a fail-safe Stop-hook script that sends Claude's response text to server-01's XTTS +endpoint. You will NOT register it in settings.json (that is a human main-session step) and +you will NOT touch server-01 or the laptop. + +## Task +Create /opt/appdata/docker/.claude/hooks/voice_tts_stop.py implementing a Stop hook that: + 1. Reads the hook JSON from stdin. + 2. GATE: if env var CLAUDE_VOICE != "1", exit 0 immediately (silent no-op). This is what + keeps every non-voice session and background-agent context silent. + 3. Extracts Claude's final response text from the hook payload. + 4. Fires the text to the server-01 XTTS endpoint as FIRE-AND-FORGET with a SHORT timeout + (<= 1.5s connect). Do not wait for audio; the laptop daemon handles playback + queueing. + 5. FAIL-SAFE: wrap everything so ANY exception, timeout, or bad payload → exit 0 silently. + The hook must NEVER raise, never block, never non-zero. Blast radius = zero. + 6. Tag the request with the session/source id so the laptop playback queue can label it. + +## Context / pre-fetched data +- The XTTS endpoint URL is not final yet (A2 produces it). Read it from an env var + VOICE_TTS_URL (default to a placeholder) so main-session can wire the real value without + editing the script. +- Hook JSON shape: confirm the current Stop-hook stdin schema against an existing hook in + /opt/appdata/docker/.claude/hooks/ before parsing (do NOT guess field names). + +## Step-by-step +1. Read one existing hook in /opt/appdata/docker/.claude/hooks/ to confirm stdin JSON shape + and the house style. +2. Write voice_tts_stop.py per the Task spec above. Use only stdlib (urllib, json, os). +3. Validate syntax: python3 -c "import ast; ast.parse(open('/opt/appdata/docker/.claude/hooks/voice_tts_stop.py').read())" +4. Test the gate: run the script with CLAUDE_VOICE unset piping a sample payload → must exit 0 + with no network call. Then with CLAUDE_VOICE=1 and VOICE_TTS_URL pointed at a dead port → + must STILL exit 0 (proving fail-safe), just no audio. +5. Do NOT edit settings.json. Report the exact settings.json Stop-hook registration snippet + for the human to paste later. + +## Output (in addition to the mandatory wrap-up) +Report: script path, ast.parse result, both gate-test outcomes, and the settings.json snippet. + +## MANDATORY WRAP-UP (required regardless of success or failure) + +Before stopping for ANY reason — task complete, error, or approaching turn limit — output this JSON as your final message. Do not stop without it. + +{ + "status": "succeeded|partially_succeeded|failed", + "actions_taken": ["action 1 — outcome", "action 2 — outcome"], + "actions_failed": ["action — reason it failed"], + "notes": "anything relevant for the next session" +} + +If you hit --max-turns before finishing, set status="partially_succeeded" and list what remains in actions_failed. + +--max-turns 15 +If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded. +``` + +--- + +## MAIN-SESSION CHECKLIST (NOT agent-izable — you + Claude in the build session) + +These cannot be background agents: they need design decisions and/or live on the LAPTOP, +which agents (running on primary) cannot reach. + +1. **Laptop audio daemon** — one local service that owns: + - Mic capture from the WorkTunes headset + PTT hotkey (hold-to-talk). + - STT client: POST captured audio to server-01 Whisper (A1's invocation) → get text. + - Injection: SSH to primary and `tmux send-keys` the text into the target Claude pane. + - Playback queue: single FIFO, consecutive/non-overlapping, source-tagged (main + each agent). + - Control-command parser: intercept `pause ` / `resume ` (and future verbs) + as commands, act on the agent's tmux instance instead of sending to Claude. +2. **tmux orchestration** — main Claude pane + one pane/window per background agent (up to 5), + named so the control parser can target them. Decide the pause mechanism (Ctrl-C interrupt + vs. cooperative signal) here — do NOT let an agent guess it. +3. **Global hook registration** — paste A3's Stop-hook snippet into settings.json; set the + voice env gate in the interactive launch wrapper (so voice = default for interactive, + silent for background/agent contexts). End-to-end test the gate. +4. **Streaming TTS** — wire sentence-by-sentence synthesis so Jarvis starts talking on sentence 1. +5. **WPM listening test** — build + run once XTTS is up (A2): synthesize passages at + 150/175/200/225/250 WPM in the Jarvis voice, comprehension check after each, record max + accurate WPM → set as default playback tempo. +6. **Training-data logging** — voice-session TEXT logged like typed sessions; audio never persisted. +``` diff --git a/agent-builder/voice_chat_build_plan.md b/agent-builder/voice_chat_build_plan.md new file mode 100644 index 0000000..441acb8 --- /dev/null +++ b/agent-builder/voice_chat_build_plan.md @@ -0,0 +1,126 @@ +# Voice Chat for Claude Code — Build Plan & Locked Decisions + +# Preflight session: 2026-07-29 (claude-config conversation) +# HARD DEADLINE: functional by 2026-08-14 (Max subscription upgrade → up to 5 background agents run regularly) +# Status: DECISIONS LOCKED. Build not yet started. + +--- + +## 0. One-paragraph summary + +A fully-local, push-to-talk voice layer over Claude Code. You speak into a Bluetooth +headset on your laptop; a laptop-side daemon ships audio to server-01's GPU for +speech-to-text (Whisper), injects the transcript into Claude Code (running in tmux on +primary) via `tmux send-keys`; when Claude finishes, a Stop hook ships the response text +to server-01 for text-to-speech (XTTS-v2, Jarvis voice), and the audio is played back on +the headset. Every interactive session is a voice session. Up to 5 background agents each +run in their own tmux instance and speak through a single, non-overlapping playback queue; +you can say "pause " to interrupt one, talk to it, then resume. + +--- + +## 1. Topology (three machines) + +| Machine | Role | Owns | +|---|---|---| +| **Laptop** (you) | Client you sit at; SSH into primary | WorkTunes Connect BT headset (mic + playback), the **audio daemon**, PTT hotkey, the **playback queue** | +| **Primary** (192.168.1.88) | Where Claude Code runs (you SSH in) | Claude Code in **tmux**, the **Stop hook**, `tmux send-keys` injection | +| **Server-01** (GPU) | Inference | **faster-whisper** (STT), **Coqui XTTS-v2** (TTS), Ollama (already there) | + +Audio makes two network hops: laptop→server-01 (STT), server-01→laptop (playback). + +--- + +## 2. LOCKED DECISIONS + +### Interaction model — phased +- **Phase A (build now, due Aug 14):** dictation + read-back wrapper, single session, PTT, Jarvis voice, PLUS the full background-agent control layer (see §3). *The control layer is Phase A, not later — confirmed 2026-07-29.* +- **Phase B (later):** hands-free conversational loop (wake word / continuous listen), streaming-TTS polish. +- **Phase C (later):** custom character voices (e.g. Setsuna / Redo of Healer) + heavy speech-pattern / dictation styling layer. + +### Engines — fully local, zero cloud, zero API cost +- **STT:** `faster-whisper` on server-01 GPU. (Aligns with `feedback_no_claude_api`, $20/mo cap, ambient-audio privacy.) +- **TTS:** **Coqui XTTS-v2** on server-01 GPU — voice-cloning capable, so Jarvis now and Setsuna later are just reference-sample swaps. (Piper rejected: fixed voices, can't clone.) + +### Voice +- **Now:** Jarvis (Iron Man). Needs a 6–30s clean reference audio sample — **USER TO PROVIDE**. +- **Later (Phase C):** anime character voices + speech-pattern styling as a text-preprocessing layer before TTS. + +### Activation +- **Push-to-talk** (hold key on laptop). Chosen for zero false triggers and because it mitigates the Bluetooth HFP/A2DP problem (mic only opens while held → playback stays high quality otherwise). + +### Default behavior +- **Every interactive session is a voice session** (voice is the default / opt-OUT, not opt-in). +- Background/agent/non-voice contexts are gated separately (see §3, §4). + +### Training data (`feedback_always_include_training_loop`) +- **Log the TEXT** of voice sessions (transcribed prompt + response) exactly as typed sessions are logged. +- **Never persist audio** — audio buffers deleted the moment STT/TTS completes. No conversation recordings. + +--- + +## 3. Background-agent control layer (PHASE A) + +This is the part that grew the scope and is required for the Aug-14 5-agent workflow. + +1. **All agent outputs are spoken** — you want to hear every agent, not just the main session. +2. **Single arbitrated playback queue** on the laptop daemon: audio plays **consecutively, never overlapping**; each chunk is **source-tagged** (which agent / main session it came from) so you know who's talking. +3. **Voice control commands** are intercepted by the daemon as commands (NOT passed to Claude): + - `pause ` → interrupt that agent's tmux instance so you can speak to/about it. + - `resume ` → resume it. + - (grammar extensible: `stop`, `skip`, `mute`, etc.) +4. **Per-agent tmux instances:** each background agent runs as its own Claude Code process in its own tmux window/pane. This is what makes independent pause/resume and clean per-source audio possible. + +Open design question for build session: how "pause" is delivered to an agent's tmux pane +(Ctrl-C interrupt vs. a cooperative pause signal) without corrupting its run — resolve in +main session, do NOT let an agent guess. + +--- + +## 4. Safety, blast radius, rollback + +- **Blast radius concern:** hooks are GLOBAL (`/home/administrator/.claude/settings.json`) — the Stop hook fires for EVERY session and background agent. A naïve synchronous hook that waits on server-01 could freeze every conversation. +- **Containment:** the TTS Stop hook must be (a) gated so it only speaks in voice contexts, (b) **async / fire-and-forget with a short timeout**, (c) **fail-safe: any error → silent `exit 0`, never blocks Claude.** +- **Rollback:** the layer is purely additive. Disable = unset the voice env gate / remove the hook registration. tmux and Claude Code are unaffected. No Vault/N8N/Coolify schema changes. +- **Blast radius after containment:** voice sessions only; worst case = no audio, never a blocked Claude. + +--- + +## 5. Hardware / latency notes + +- **WorkTunes Connect (Bluetooth):** HFP/A2DP mutual exclusion — mic + high-quality playback can't coexist; PTT mitigates. Internal mic fine for commands, NOT good enough for recording a reference voice sample. +- **Headset upgrade (DEFERRED):** wired USB headset sidesteps HFP/A2DP entirely (clean mic + full-quality playback simultaneously). Recommend when ready; not a build blocker. +- **Latency budget (LAN + GPU):** ~1s before Claude thinks (BT capture + STT) + ~1–3s after (XTTS), dominated by Claude's own generation. Mitigate with **sentence-by-sentence streaming TTS** (Jarvis starts talking on sentence 1). + +--- + +## 6. Deliverables of the preflight session + +- [x] Locked grill-me decisions (this document). +- [ ] **WPM listening test** — script that synthesizes passages at 150/175/200/225/250 WPM in the Jarvis voice, asks a comprehension check after each, and reports your max accurate listening WPM → becomes the default TTS playback tempo. Doubles as first end-to-end server-01→laptop audio check. (Build + run once XTTS stands up.) +- [ ] **Build-plan agent prompts** (see §8) — bounded server-01 prompts + main-session checklist. + +## 7. Open items / user tasks + +- **Jarvis reference audio** (6–30s clean) — user provides. +- **server-01 GPU/VRAM audit** — can XTTS-v2 + Whisper + Ollama coexist in VRAM? (First build step / first agent task.) +- **Headset upgrade** — deferred decision. + +--- + +## 8. Build execution shape (NOT one agent) + +Per `playbook_background_agent_prompts.md` (agents = bounded ≤15 tool calls, defined +completion, no design) and the topology (agents run on primary, CANNOT touch the laptop), +the build splits into: + +- **Bounded background-agent prompts (server-01 only):** + - A1 — install + smoke-test `faster-whisper` on server-01 GPU; report VRAM headroom. + - A2 — install + smoke-test XTTS-v2; clone from a provided Jarvis sample; emit a test WAV. + - A3 — write + validate the fail-safe async Stop-hook TTS script (does NOT register it globally without confirmation). +- **Main-session work (you + Claude, cannot be agent-ized):** + - Laptop audio daemon (mic, playback queue, PTT, control-command parser). + - tmux injection wiring + per-agent tmux orchestration. + - Global hook registration + gating + end-to-end integration test. + +Agent prompts to be written into `voice_chat_agent_prompts.md` (alongside `agent_prompts.md`).