Backtalk6858/claude-projects
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(prompts): save 2026-09-24/25 agent prompts verbatim (O, P, media-pipeline/, infra/, wireguard/)

Browse Source 
13 prompts recovered verbatim from the session transcript (they had been spawned inline without being
saved first) + new O (DIY WireGuard design), P (login-layer alternatives), MP-15a density scan,
MP-15 v2 hybrid picker, MP-15 v1 (superseded, never spawned), wireguard/W1_build_prep. README indexes
the new project subfolders. Rule added to playbook_background_agent_prompts: save first, then spawn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Backtalk6858
2026-09-25 19:22:04 -05:00
parent 351c86fde2
commit 3cab46b9bb
20 changed files with 1116 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
claude-config/config/prompts/O_diy-wireguard-design.md
+44
View File
@@ -0,0 +1,44 @@
# O — DIY WireGuard design (personal_projects #274) — DESIGN ONLY
Tracks: personal_projects #274. Spawn from: the media-downloader-local / infrastructure conversation.
Output: `/opt/appdata/docker/research/diy_wireguard_design.md`. Read-only; no installs. Written 2026-09-25.
---
You are a bounded background DESIGN agent for personal_projects #274 "DIY WireGuard (wg-easy) for owner private access". Budget: --max-turns 30. Output = one design doc ready for an owner grill-me; you change nothing.
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with status=partially_succeeded.
HARD RULES
- READ-ONLY on both hosts (primary = this host; server-01 = ssh administrator@192.168.1.90): docker ps / inspect (image, ports, networks, labels only — NEVER .Config.Env), reading compose/config files WITHOUT printing secret values, `ip -br addr`, `ss -lntup` (no sudo), reading /etc/hosts. No installs, no container starts, no config edits, no DNS/router/Cloudflare changes, no Postgres writes, no git add/commit/push, no sudo, no notifications. Mask secrets. If a hook blocks something, stop with a partial wrap-up.
- Web research allowed (official docs/GitHub first; note versions/dates). Mark unconfirmed claims UNVERIFIED.
OWNER DECISIONS / CONSTRAINTS (2026-09-25)
- Goal: owner-owned remote access for the OWNER's devices: laptop, tablet, and phone (Android 9 → Twingate unsupported (needs 10+), WireGuard app supports Android 5+). Partners stay on Twingate (already working). Self-hosted NetBird is RETIRED (its control plane cannot live behind the Cloudflare free tunnel; Headscale has the same blocker per its docs). cloudflared stays as-is FOR NOW (all 21 hostnames); later the admin sites get pulled off the tunnel and reached over WireGuard.
- $0 cost is the top priority. NO inbound TCP ports. ONE inbound UDP port for WireGuard is acceptable because WireGuard is silent to unauthenticated packets. UDP 3479 (NetBird STUN) stays closed.
- Security principles to honour: P5 (unnecessary exposure eliminated, not protected), P9 (zero-trust internal: network segmentation, don't dump new services on the flat `coolify` network by default), secrets only in Vault (AppRole), never on disk in plaintext configs where avoidable. Primary is RAM-constrained (32 GB, oversubscribed) — anything new must be light.
- Public IP is DYNAMIC (66.164.11.87 on 09-17, 66.164.12.179 on 09-25). DNS for reverseproxyserver.net is on Cloudflare and stays there (DNS-only records are free; a scoped Zone:DNS:Edit API token is needed for DDNS and for DNS-01 certificates — not created yet).
- Traefik (container `coolify-proxy`) listens only on 127.0.0.1:80 (plain HTTP; TLS is terminated at Cloudflare's edge today). Services live on the docker network(s) — inspect coolify-proxy's networks and the 172.16.16.0/20 range. Precedent: Gitea is reached locally by pinning its hostname in /etc/hosts (find how, read-only).
- A separate agent (prompt P) is researching a replacement LOGIN layer for Authelia (owner distrusts Authelia after many forward-auth breakages; Authentik failed in LibreWolf). Your design must leave a clean slot for "login layer in front of admin sites" without choosing it.
- The owner wants to keep a Twingate account on the laptop as a BACKUP path into the network if WireGuard breaks.
DESIGN QUESTIONS TO ANSWER (recommend a default for each, with reasons and trade-offs)
1. wg-easy (container, web UI, check current major version features: 2FA on admin UI, per-client config, QR codes, metrics) vs plain wg-quick on the host vs other OSS UIs. Container vs host kernel module; network mode (host vs bridge) and P9 placement; RAM.
2. Port: which UDP port (non-default vs 51820), router forward (owner-managed; owner will do it), and verification that it is silent (how to test from outside safely).
3. DDNS for the dynamic IP: options (a small container like a maintained cloudflare-ddns image vs a systemd script using the CF API), update interval, the WireGuard client endpoint re-resolution behaviour (Android/Linux clients when the IP changes; PersistentKeepalive; how long an outage lasts), where the CF token lives (Vault) and how it is injected (read how services get secrets today: look for `*-secretspec-resolver` units and one deploy/resolver.sh, read-only).
4. Client DNS / split-horizon: how the owner's devices resolve *.reverseproxyserver.net to the LOCAL Traefik when on WireGuard (options: WireGuard DNS= pointing at a small resolver (dnsmasq/CoreDNS/Unbound/AdGuard — check if any DNS server already runs on either host), per-device hosts entries, Cloudflare private records — note security of each). Must not break when the device is off-VPN.
5. Local HTTPS: today Traefik has no local TLS. Options: DNS-01 wildcard *.reverseproxyserver.net via the CF token (Traefik certresolver) on a LAN/VPN-only 443 entrypoint bound to the VPN/LAN interface (NOT forwarded on the router), vs plain HTTP inside the tunnel. Recommend; note HSTS risks and the Traefik change's blast radius (shared by 31 services).
6. Access control per device (ACLs): allowed destinations per peer (Traefik only? LAN 192.168.1.0/24? SSH to primary 22? server-01?), enforced where (wg-easy's own settings vs nftables/DOCKER-USER rules); phone = narrow, laptop = admin.
7. Keys: generation, backup to Vault (path proposal `secret/wireguard/...`), revocation, rotation cadence, lost-phone procedure.
8. Failure modes + monitoring: server down, IP change, key compromise; health check + ntfy alert; interaction with the #273 image auto-updater (tier for wg-easy/ddns images).
9. Migration plan: phases with verification + rollback — W0 owner prep (CF token → Vault, router UDP forward), W1 WireGuard up + laptop over LAN test, W2 DDNS + off-LAN test (phone on cellular), W3 split-DNS + local HTTPS, W4 ACLs, W5 (later, gated on the login-layer decision) pull admin hostnames off cloudflared one group at a time. Also: stopping NetBird (owner runs `docker stop netbird-server netbird-dashboard`; keep data) — note what else references NetBird (Traefik labels, CF tunnel route, Authelia dead config) for later cleanup.
10. Effort estimate per phase (hours) and an honest difficulty rating.
OUTPUT: write /opt/appdata/docker/research/diy_wireguard_design.md: Summary (plain language) / Constraints / Current-state facts found / Q1–Q10 with recommendations / Phased plan / Risks / Owner questions (only real decisions) / Sources.
SCOPE ALLOWLIST: create that one file; the context.md append. Nothing else.
PERSIST BEFORE YOU FINISH
- Append (one `cat >>` heredoc; append only — other agents may write concurrently) "## #274 DIY WireGuard design — 2026-09-25 (background agent)" to "/opt/appdata/docker/Machines/infrastructure general questions/.claude/context.md": What was done / Recommendations / Next step.
- Run: python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5
- FINAL message = wrap-up JSON only, always:
{"status":"succeeded|partially_succeeded|failed","project":"diy-wireguard #274","actions_taken":[],"actions_failed":[],"files_touched":[],"containers_restarted":[],"unverified_claims":[],"recommendations":{"server":"","port":"","ddns":"","split_dns":"","local_https":"","acls":"","keys":""},"phases":[],"effort_hours":null,"difficulty":"","owner_questions":[],"report":"","next_step":"","notes":""}
claude-config/config/prompts/P_login-layer-alternatives.md
+37
View File
@@ -0,0 +1,37 @@
# P — Login layer alternatives to Authelia — RESEARCH ONLY
Tracks: infrastructure (login layer; pairs with #274 DIY WireGuard). Spawn from: the media-downloader-local / infrastructure conversation.
Output: `/opt/appdata/docker/research/login_layer_alternatives.md`. Read-only. Written 2026-09-25.
---
You are a bounded background RESEARCH agent (infrastructure; login layer for the owner's admin sites). Budget: --max-turns 25. Output = one research doc; you change nothing.
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with status=partially_succeeded.
HARD RULES
- READ-ONLY on the host: you may read (never print secret values) Authelia's configuration and Traefik labels/middlewares to learn how forward-auth is wired today (find the authelia compose dir under /opt/appdata/docker/docker-compose/; redact secrets, keys, hashes, emails). No container starts/restarts, no config edits, no Postgres writes, no git add/commit/push, no sudo, no notifications. If a hook blocks something, stop with a partial wrap-up.
- Web research allowed (official docs/GitHub first; note versions/dates). Mark unconfirmed claims UNVERIFIED.
OWNER REQUEST (2026-09-25)
"I'd like a login screen, but we've run into so many issues where Authelia hasn't worked with a service that I don't really trust it anymore, and Authentik won't work either — the only fix we found was switching web browsers because LibreWolf's extreme privacy settings broke Authentik's web UI. Find an alternative to Authelia if possible."
CONTEXT (verified history)
- Authelia (current SSO/forward-auth, with 2FA; privacyIDEA also runs) breakages we hit: (1) Traefik forwardAuth blocked apps' own AJAX/API calls (no session cookie) → WebUI logins failed; fixed globally with a wildcard API bypass rule, but new apps with non-standard API paths (/auth, /rpc, /json, /api/v2) keep breaking; (2) Authelia refuses to set cookies over plain HTTP → needed an X-Forwarded-Proto: https headers middleware chain because TLS terminates at Cloudflare and Traefik is plain HTTP; (3) Authelia exits on SIGHUP and needs a restart after config edits; (4) as an OIDC provider for NetBird it needed CORS, opaque-vs-JWT token fixes and a name-claim policy and still hung (abandoned). Services with their own strong auth/2FA (Gitea, Jellyfin, privacyIDEA) bypass Authelia (principle P8: don't double-stack 2FA).
- Authentik: deployed once; its web UI login broke in LibreWolf/Firefox with strict privacy settings (resistFingerprinting / strict tracking protection / cookie isolation suspected); only workaround was another browser/profile — unacceptable.
- The owner uses LibreWolf with extreme privacy settings. HARD REQUIREMENT: the login page must work in LibreWolf defaults+strict. Identify from docs/issues which JS/cookie/WebAuthn/fingerprinting features each candidate's login UI depends on and whether known LibreWolf/Firefox-RFP issues exist.
- Upcoming network model: the owner reaches admin sites over DIY WireGuard (#274, designed in parallel by prompt O); partners over Twingate; cloudflared keeps public sites for now. Traefik (coolify-proxy) is the reverse proxy for 31 services, plain HTTP locally today (local HTTPS via DNS-01 is planned).
- Primary host is RAM-constrained — lighter is better. $0 / open source only. Principles: P5 (don't expose what needn't be), P8, P9 zero-trust internal.
TASK
1. Root-cause honestly: which of the Authelia breakages are INHERENT to the forward-auth pattern (any replacement would hit them) vs Authelia-specific. State this plainly — the owner needs to know whether switching products fixes the pain.
2. Evaluate candidates (at minimum): TinyAuth, Pocket ID (passkey-only OIDC; pair with a forward-auth shim like TinyAuth or oauth2-proxy), oauth2-proxy, Kanidm, VoidAuth, Zitadel, Keycloak, Authelia (stay, but simplify), and "no SSO: per-service native auth + network-level access (WireGuard) + Traefik basicAuth only where a service has no auth". For each: Traefik forwardAuth support, 2FA/TOTP/passkeys, LibreWolf/Firefox-strict compatibility (evidence), RAM footprint, config style (file/UI), API-path bypass handling, maturity/maintenance/licence, OIDC provider ability (for future apps), migration effort from Authelia.
3. Recommend ONE approach (possibly "fewer login layers" given WireGuard) + a fallback, with a migration plan (phases, verification incl. a LibreWolf test checklist the OWNER runs, rollback to Authelia), and which services get the login layer vs native auth.
4. Write /opt/appdata/docker/research/login_layer_alternatives.md: Verdict (plain language) / Inherent vs Authelia-specific problems / Requirements / Comparison table / Recommendation + migration / LibreWolf test checklist / Risks / Owner questions (real decisions only) / Sources.
SCOPE ALLOWLIST: create that one file; the context.md append. Nothing else.
PERSIST BEFORE YOU FINISH
- Append (one `cat >>` heredoc; append only) "## login-layer alternatives research — 2026-09-25 (background agent)" to "/opt/appdata/docker/Machines/infrastructure general questions/.claude/context.md": What was done / Recommendation / Next step.
- Run: python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5
- FINAL message = wrap-up JSON only, always:
{"status":"succeeded|partially_succeeded|failed","project":"login-layer (Authelia replacement)","actions_taken":[],"actions_failed":[],"files_touched":[],"containers_restarted":[],"unverified_claims":[],"inherent_problems":[],"authelia_specific_problems":[],"candidates":[{"name":"","forward_auth":null,"librewolf_ok":"","ram":"","verdict":""}],"recommendation":"","fallback":"","owner_questions":[],"report":"","next_step":"","notes":""}
claude-config/config/prompts/README.md
+9
View File
@@ -19,6 +19,8 @@ Canonical copies live here; mirrored to Obsidian `Resources/Prompts/`. Each prom
| K | [K_netbird-twingate-preflight.md](K_netbird-twingate-preflight.md) | primary-server hub (netbird-twingate) | claude-config | `research/{twingate-deploy-runbook,netbird-selfhost-buildplan,partner-access-audit-austin-hailee}.md` — **read-only preflight** |
| L | [L_twingate-partner-swap-deploy.md](L_twingate-partner-swap-deploy.md) | primary-server hub (netbird-twingate) | claude-config | Twingate connector deploy + Austin→Hailee swap — **⚠ EXECUTION/mutating, not research-only; run after K + owner review** |
| M | [M_hailee-nextcloud-onboard.md](M_hailee-nextcloud-onboard.md) | primary-server hub (netbird-twingate) | claude-config | Onboard Hailee into Nextcloud (additive only) — **⚠ EXECUTION/mutating; run AFTER K; needs Hailee username+email filled in first** |
| O | [O_diy-wireguard-design.md](O_diy-wireguard-design.md) | personal_projects 274 (DIY WireGuard; replaces parked NetBird #258) | media-downloader-local / infra conversation | `/opt/appdata/docker/research/diy_wireguard_design.md` — design-only, read-only (2026-09-25) |
| P | [P_login-layer-alternatives.md](P_login-layer-alternatives.md) | infra login layer (Authelia replacement; pairs with 274) | media-downloader-local / infra conversation | `/opt/appdata/docker/research/login_layer_alternatives.md` — research-only; LibreWolf hard requirement (2026-09-25) |
**Bootstrap rule (user, 2026-09-09):** ad budget is $0. D/E/F must each give a $0 path to first sales and a time-to-first-sale ranking input; the business that bootstraps fastest launches first and its profits fund ads for the next. A business with no $0 path is PAUSED.
@@ -28,3 +30,10 @@ follow-up grill-me before the next spawn. Works on Fable 5.1 or Opus 4.8 unchang
**Grill-me that shaped A and B:** `decisions/DECISIONS.md` 2026-09-08 "AI-harness layer" entry
(Obsidian `Resources/Grill-Me/2026-09-08 — AI Harness Layer.md`).
## Project subfolders (added 2026-09-25)
Execution/build/design prompts that belong to an ongoing project set live in a subfolder, saved **verbatim as spawned**:
- `media-pipeline/` — MP-10 build, AV-CHECK, AV-FIX, VAAPI QP calibration, VAAPI build (2026-09-24); MP-18 build, MP-19 design, MP-19b design, MP-19b R2 build (2026-09-25). Index/decisions: memory `playbook_media_pipeline_phases.md`.
- `infra/` — NTFY-272 hardening (2026-09-24); #273 image-autoupdate research + P1a pin proof, cloudflared replacement research (2026-09-25). #273 P2 prompt lives verbatim in memory `playbook_image_autoupdate_phases.md`.
The 13 files above were recovered from the session transcript on 2026-09-25 because they had been spawned inline without being saved first.
- `wireguard/` — #274 home-access tunnel phase prompts: `W1_build_prep.md` (2026-09-25; write files only, no deploy). Design: `/opt/appdata/docker/research/diy_wireguard_design.md`.
claude-config/config/prompts/infra/273_P1a_pin_proof_policy.md
+32
View File
@@ -0,0 +1,32 @@
# 273 P1a pin proof policy — exact prompt as spawned (2026-09-25)
Tracks: personal_projects 273. Recovered verbatim from the session transcript 2026-09-25 (it was spawned inline and not saved at the time).
---
You are a bounded background agent for personal_projects #273 (image auto-updater), PHASE P1a. Budget: --max-turns 25.
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with status=partially_succeeded.
HARD RULES
- PROD IS READ-ONLY: on primary and server-01 you may only read (`docker ps`, `docker inspect` filtered to image/labels/created/restart fields — NEVER .Config.Env, `docker image inspect`, reading compose files, `systemctl cat`, `git diff`). NEVER edit any prod compose file — every `*-secretspec-resolver` timer runs `compose up -d` every 15 min, so an edit to a prod `image:` line auto-deploys. NEVER restart/pull/recreate prod containers.
- The ONLY place you may run containers is server-01 (ssh administrator@192.168.1.90) inside a NEW scratch dir `~/p1a-scratch` with your own throwaway compose project (project name `p1a-test`). Use a tiny image (e.g. `busybox` or `alpine` with a sleep command) — nothing else. Tear it down (`docker compose -p p1a-test down`) and delete the dir at the end. Do not touch any other container on server-01 (it runs prod services + sandboxes).
- No git add/commit/push, no Postgres writes, no sudo, no notifications. Mask secrets. If a hook blocks something, stop with a partial wrap-up.
- Issue each mutating command as its own tool call; verify separately.
CONTEXT
- Design: /opt/appdata/docker/research/image_autoupdate_design.md (READ §2 inventory, §3 tiers, §6 design, §7 P1, §8 risks). Phase plan + owner decisions: /home/administrator/.claude/projects/-opt-appdata-docker/memory/playbook_image_autoupdate_phases.md (READ).
- Owner tier decisions: hybrid N-1 (newest once >=14 d old else previous patch); LATEST tier = newest stable >=24 h; qbittorrent + qbittorrent-private move to LATEST; autobrr/qui/grafana/prometheus/cadvisor stay N-1; custom gitea.local/backtalk6858/* images excluded; ollama-fixed (server-01 local build) excluded; coolify-db to be removed (mark `remove`).
- The docker compose version on each host: check `docker compose version` on BOTH hosts and use the same major/minor behaviour in your test (note it).
TASK
1. PROOF (the question the whole P1 depends on): does `docker compose up -d` RECREATE a running container when the service's `image:` string changes from `<repo>:latest` to `<repo>:<exact-tag>@sha256:<digest>` (or `<repo>:<exact-tag>`) that resolves to the SAME local image ID? In ~/p1a-scratch on server-01: pull e.g. `busybox:1.36.1` (and tag it locally as needed so `:latest`-style and exact refs point to the same ID — or use a real image where `:latest` currently equals a known version), `up -d` with ref A, record container ID + Created, change only the image line to ref B (same image ID), `up -d` again, record whether the container was recreated (ID/Created changed) and what compose printed. Test both variants: (a) `:tag` → `:exact-tag` same ID, (b) → `tag@digest`. Also test `docker compose up -d --no-recreate` behaviour for reference. State the verdict plainly: RECREATES or DOES NOT RECREATE.
2. POLICY DRAFT: write /opt/appdata/docker/image-policy.draft.yml (NEW file, clearly marked DRAFT at top) with one entry per managed service per the design §6.1 fields (compose path, service key, image repo, current ref, current RUNNING version — from image label org.opencontainers.image.version or a digest→tag match against the registry (read-only registry/Hub API calls are fine), tier, versioning, auto (LATEST→notify, N-1→notify for now), deploy method (resolver unit name or `compose`), migrating:true for nextcloud/gitea/authelia/vault/postgres/n8n). Mark unresolvable versions `UNRESOLVED` with the reason.
3. PROPOSED PIN TABLE: write /opt/appdata/docker/research/image_pin_plan_P1.md: the recreate verdict + evidence; a table per host of service → current ref → proposed pinned ref (tag@digest) → tier → deploy method → risk (low/med/high, migrating?) → suggested order; and a recommended P1b procedure given the verdict (if RECREATES: one service at a time in an owner-watched window, low-risk first, with the exact verification per service; if not: still batch in small groups). List the services WITHOUT a resolver (netbird, traefik/coolify-proxy, twingate, redis, filebot-cli, …) and what deploy path each needs. Note the custom vault image FROM line and jenkins FROM line (read the Dockerfiles; do not edit).
SCOPE ALLOWLIST: create /opt/appdata/docker/image-policy.draft.yml and /opt/appdata/docker/research/image_pin_plan_P1.md; server-01 ~/p1a-scratch + project p1a-test (create and remove); the context.md append. Nothing else.
PERSIST BEFORE YOU FINISH
- Append (one `cat >>` heredoc; append only) "## #273 P1a — 2026-09-25 (background agent)" to "/opt/appdata/docker/Machines/infrastructure general questions/.claude/context.md": What was done / Recreate verdict / Next step.
- Run: python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5
- FINAL message = wrap-up JSON only, always:
{"status":"succeeded|partially_succeeded|failed","project":"image-autoupdate #273","phase":"P1a","actions_taken":[],"actions_failed":[],"files_touched":[],"containers_restarted":[],"unverified_claims":[],"compose_versions":{"primary":"","server01":""},"recreate_verdict":"RECREATES|DOES_NOT_RECREATE","evidence":{"variant_a":"","variant_b":"","no_recreate_flag":""},"services_in_policy":null,"unresolved_versions":[],"no_resolver_services":[],"p1b_procedure_summary":"","next_step":"","notes":""}
claude-config/config/prompts/infra/273_image_autoupdate_research.md
+40
View File
@@ -0,0 +1,40 @@
# 273 image autoupdate research — exact prompt as spawned (2026-09-25)
Tracks: personal_projects 273. Recovered verbatim from the session transcript 2026-09-25 (it was spawned inline and not saved at the time).
---
You are a bounded background RESEARCH agent (personal_projects #273, infrastructure). Budget: --max-turns 25. Output = one research/design doc; you change nothing.
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with status=partially_succeeded.
HARD RULES
- READ-ONLY on both hosts: `docker ps`, `docker inspect`, `docker image ls`, reading compose files. NEVER pull, restart, recreate, stop, or exec mutating commands in any container. No deploys, no compose edits, no Postgres writes, no git add/commit/push, no sudo, no notifications.
- Never print secrets: when reading compose/env files, do not echo env values; `docker inspect` output must be filtered to image/labels/restart policy only (use python to extract .Config.Image, .Image, .Config.Labels keys, .HostConfig.RestartPolicy — never .Config.Env).
- If a hook blocks an action, do not work around it: partial wrap-up.
- Web research is allowed (WebSearch/WebFetch). Prefer each tool's OFFICIAL docs/GitHub; note the date/version you read. Mark anything you could not confirm as UNVERIFIED.
CONTEXT (verified by the main session 2026-09-25)
- Two hosts: primary (this host; AMD Ryzen 7 5800H, 32 GB, RAM-constrained/oversubscribed — anything new here must be tiny) and server-01 (ssh administrator@192.168.1.90; 24 cores; runs some offloaded prod services + sandboxes).
- Watchtower was RETIRED 2026-07-22; NOTHING auto-updates images today (no watchtower container on either host). Images update only when someone pulls manually. Most compose files use `:latest`.
- Compose files live under /opt/appdata/docker/docker-compose/<service>/ (git repo /opt/appdata/docker). Many prod services get secrets from Vault via SecretSpec: they are (re)created ONLY by `sudo systemctl start <svc>-secretspec-resolver.service` — a plain `docker compose up -d` from a shell blanks their Vault-injected secrets. Any updater that recreates containers itself (Watchtower-style) would bypass that path — this is a KEY constraint to analyse (does the tool recreate containers with the same env? does it break secretspec services? can it instead only change the pinned tag in git and let the owner/resolver deploy?). Read one resolver unit + its script to understand the path: `systemctl cat media-api-secretspec-resolver.service` (read-only) and the script it runs.
- Custom images are built and pushed to Gitea (`gitea.local/backtalk6858/*`) with SemVer + latest; they follow their own versioning and are OUT of scope for auto-update.
- Coolify is retired; Jenkins exists but no deploy pipeline is built. ntfy is the notification system (self-hosted).
- Owner rules: containers are never restarted by Claude/agents (the owner restarts). An automated updater is a SYSTEM decision the owner is making now — design it so every restart is either owner-approved or policy-approved, with notification.
- OWNER POLICY (the requirement):
(1) Third-party images stay ONE VERSION BEHIND the newest release (to avoid day-0 bugs/crashes).
(2) Security-critical / internet-facing services track the LATEST release (patches matter more than stability) — e.g. NetBird (netbirdio/netbird-server, netbirdio/dashboard), Traefik (coolify-proxy), cloudflared, Vaultwarden, Authelia, Vault, Nextcloud, anything reachable from outside. Propose the exact list from the inventory with a one-line reason each.
(3) Custom Gitea images excluded.
TASK
1. Inventory (read-only): every running container on both hosts → image ref, tag style (latest / semver / digest), and whether its compose dir uses a secretspec resolver (look for `<svc>-secretspec-resolver.service` units via `systemctl list-units --all '*secretspec*'` and compose dirs). Classify each into tier N-1, tier LATEST, or EXCLUDED.
2. Research candidate tools/approaches (at minimum): Renovate (self-hosted, against the Gitea repo; `minimumReleaseAge`, versioning, digest pinning, automerge), What's Up Docker (WUD), diun, the maintained Watchtower fork(s) (check whether containrrr/watchtower is archived and which fork is maintained), Komodo or similar, and a small custom resolver (registry API: list tags, semver-sort, pick newest-minus-one). For each: can it express "one version behind" (true N-1 vs. a minimum-age delay — explain the difference and which better meets the owner's intent), per-service tiers, handles non-semver tags, rollback on a failed healthcheck, ntfy notifications, how it deploys (recreates containers itself vs. edits pins in git), compatibility with the secretspec resolver path, resource cost on primary, maintenance status/licence.
3. Recommend ONE design: components, where it runs (primary vs server-01), the flow (detect → decide tier → pin change → deploy via the resolver/owner approval → health check → rollback → ntfy), how the N-1 tag is resolved, schedule, and a phased build plan (phase prompts outline: P1 inventory+pin to explicit versions, P2 detector+notifications, P3 automated pin bumps, P4 automated deploy with rollback — each with a verification). Include what changes in the Docker Template / tag rule (`:latest` → explicit pinned versions?).
4. Write /opt/appdata/docker/research/image_autoupdate_design.md: Summary/verdict up top / Inventory table / Tier assignment / Tool comparison table / Recommendation / Flow diagram (text) / Phased build plan / Risks / Owner questions (only real decisions). UNVERIFIED markers where applicable.
SCOPE ALLOWLIST: create /opt/appdata/docker/research/image_autoupdate_design.md; the context.md append below. Nothing else.
PERSIST BEFORE YOU FINISH
- Append (one `cat >>` heredoc; append only — other agents may write concurrently) a dated block "## #273 image auto-updater research — 2026-09-25 (background agent)" to "/opt/appdata/docker/Machines/infrastructure general questions/.claude/context.md": What was done / Recommendation / Next step.
- Run: python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5
- Your FINAL message is the wrap-up JSON only, always:
{"status":"succeeded|partially_succeeded|failed","project":"image-autoupdate #273","actions_taken":[],"actions_failed":[],"files_touched":[],"containers_restarted":[],"unverified_claims":[],"inventory_counts":{"primary":null,"server01":null,"tier_latest":[],"tier_n_minus_1":null,"excluded":[]},"secretspec_services":[],"tools_compared":[],"recommendation":"","n_minus_1_mechanism":"","report":"","owner_questions":[],"next_step":"","notes":""}
claude-config/config/prompts/infra/NTFY-272_hardening.md
+59
View File
@@ -0,0 +1,59 @@
# NTFY-272 hardening — exact prompt as spawned (2026-09-24)
Tracks: personal_projects 272. Recovered verbatim from the session transcript 2026-09-25 (it was spawned inline and not saved at the time).
---
You are a bounded background agent for personal_projects #272 (ntfy hardening). Budget: --max-turns 25.
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up JSON with
status=partially_succeeded.
HARD RULES
- Owner-authorized (2026-09-24) to change ntfy USERS/ACCESS via `docker exec ntfy-ntfy-1 ntfy user|access ...` CLI only.
NEVER restart, stop or recreate ANY container (ntfy, media-api or other) — restarts are the owner's; put the exact
command in your output. Never change ntfy server config/compose.
- TOKENS NEVER APPEAR IN ANY OUTPUT, FILE, OR COMMAND LINE YOU WRITE. You may NOT create tokens or write Vault yourself:
any step that creates a token and stores it in Vault goes to the owner via the relay below.
- Owner-command relay: end your run with a final message headed exactly `⏸ OWNER-COMMAND-REQUEST` containing host:, command:
(one copy-pasteable command per line, no placeholders, no secret values — the command must generate the token and write
it to Vault in one pipeline without echoing it), why:, paste_back: (only non-secret confirmation, e.g. Vault version
number), resume_at: — then stop. Max 3 relays.
- NO git add/commit/push, NO DB writes, mask secrets everywhere, no sudo. Blocked → partial wrap-up, never work around.
- Other agents run in parallel on media_pipeline code and the server-01 sandbox — do not touch those files.
ALLOWED FILES: /home/administrator/.claude/projects/-opt-appdata-docker/memory/playbook_ntfy_users_topics.md (edit),
the qbit mount monitor script found in step 3 (edit only its topic/token-variable wiring), and the persist targets.
PERSIST: append a dated "## NTFY-272 — <date> (background agent)" block (What was done / Decisions / State / Next) to
/opt/appdata/docker/docker-compose/media-downloader-local/.claude/context.md (single append); run
python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5.
FINAL message = wrap-up JSON only (unless pausing with the relay): status, project ("ntfy_272"), actions_taken[],
actions_failed[], files_touched[], containers_restarted[] (must be empty), unverified_claims[], next_step, notes + extra fields.
==========================================================================================
Context (from research /opt/appdata/docker/research/notifarr_vs_ntfy.md §1 + §6 — read them): ntfy server container
ntfy-ntfy-1, deny-all default, per-topic write-only bot users, reader account Smoked5003. The OWNER already moved the phone
off the admin account (done). Remaining fixes:
1. media-api publishes to `media-api-notifications` but no user has write access there → probably 403. Confirm read-only
(ntfy access list; which ntfy user media-api's current token belongs to — determine via `ntfy token list` output for users,
printing only user names/labels, never token values; media-api's env var is NTFY_TOKEN, sourced by secretspec from Vault
path secret/data/secretspec/media-api/default/NTFY_TOKEN). Then: create user `media-api-bot` (random password generated
inside the command, never printed; bots only use tokens), grant `media-api-bot media-api-notifications write-only`,
grant `Smoked5003 media-api-notifications read-only`. Verify with `ntfy access`.
2. Token for media-api-bot → owner relay: the command must run `docker exec ntfy-ntfy-1 ntfy token add -l media-api media-api-bot`,
capture the tk_ value, and write it to Vault at secret/data/secretspec/media-api/default/NTFY_TOKEN (KV v2, field name must
match how secretspec reads it — read /opt/appdata/docker/docker-compose/media-api/secretspec.toml and the secretspec
playbook /home/administrator/.claude/projects/-opt-appdata-docker/memory/playbook_secretspec_env_resolution.md to get the
exact path/field) using Vault AppRole auth per
/home/administrator/.claude/projects/-opt-appdata-docker/memory/playbook_vault_token_rotation.md Step 0 (Vault
192.168.1.88:8200; AppRole only — NEVER the root token). Also give the owner the media-api restart command. Old token:
list it for revocation in next_step only after the owner confirms media-api works.
3. The qBittorrent mount monitor (monitor-qbit-mount.sh — find it with grep -rl under /opt/appdata/docker) publishes to
media-transcoder-notifications. Give it its own bot `qbit-mount-bot` + topic `qbit-mount-notifications`
(bot write-only, Smoked5003 read-only), token via the SAME relay (batch both tokens into ONE relay if possible; store at
secret/data/ntfy/qbit-mount-bot field token), and change the script to use the new topic + read its token the same way
it reads the current one (do not hard-code any secret). If the script runs from cron/systemd, say how it picks the change up.
4. Rewrite the playbook's stale parts: registry (add media-api-bot, qbit-mount-bot, secrets-proxy-bot, current Smoked5003
grants; fix the IP conflicts noted in the research §6), and REPLACE every Vault step that pulls the root token from
Bitwarden with the AppRole method (reference playbook_vault_token_rotation Step 0). Keep its update instructions.
5. Verification you can do without secrets: `ntfy access` shows the new grants. The owner verifies delivery after restart
(tell them: a media-api notification should arrive on the phone).
Extra wrap-up fields: {"media_api_403_confirmed":null,"users_created":[],"grants":[],"relay_needed":true,
"playbook_sections_rewritten":[],"owner_restart_cmds":[],"revoke_after_confirm":[]}
claude-config/config/prompts/infra/cloudflared_replacement_research.md
+40
View File
@@ -0,0 +1,40 @@
# cloudflared replacement research — exact prompt as spawned (2026-09-25)
Tracks: personal_projects 258/274. Recovered verbatim from the session transcript 2026-09-25 (it was spawned inline and not saved at the time).
---
You are a bounded background RESEARCH agent (infrastructure; related to personal_projects #258 NetBird). Budget: --max-turns 25. Output = one research doc; you change nothing.
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with status=partially_succeeded.
HARD RULES
- READ-ONLY on both hosts. No container restarts/pulls/deploys, no compose or config edits, no DNS/Cloudflare changes, no Postgres writes, no git add/commit/push, no sudo, no notifications. Never print secrets (the cloudflared container carries TUNNEL_TOKEN — never print env values; use docker inspect only for image/cmd/ports/labels).
- Web research allowed (WebSearch/WebFetch). Prefer official docs/GitHub; note versions/dates read. Mark unconfirmed claims UNVERIFIED.
- If a hook blocks an action, do not work around it: partial wrap-up.
OWNER QUESTION (2026-09-25)
"Analyze what we are using cloudflared for, and what we still plan to use it for after NetBird is fully deployed (the plan is to leave NetBird's control plane on cloudflared so cloudflared acts as its front door). Then find a FREE, OPEN-SOURCE alternative that accomplishes that, so we can FULLY RETIRE cloudflared — I'm sick of the Cloudflare free account's limitations."
VERIFIED CONTEXT
- Primary host runs a `cloudflared` container (remote-managed tunnel via token; cmd `tunnel --no-autoupdate run --protocol http2`) that forwards the public hostnames `*.reverseproxyserver.net` to Traefik (`coolify-proxy`, which only listens on 127.0.0.1:80 — plain HTTP; TLS terminates at Cloudflare's edge). Traefik's certs: see how it is configured today (read-only: find its static config/labels) — prior notes say http-01 through cloudflared.
- Inventory of the 21 public hostnames + cutover categories: /home/administrator/Desktop/claude/claude-config/research/cloudflared-tunnel-inventory_2026-09-17.md (READ IT). NetBird design + decisions: /home/administrator/Desktop/claude/claude-config/research/netbird-design_2026-09-17.md and /home/administrator/Desktop/claude/claude-config/config/prompts/N2_netbird_primary_deploy.md (B-refined decision: control plane through cloudflared, only UDP STUN opened; "Option A" = direct FQDN via Traefik with TCP 443 open, DNS-01 wildcard via a Cloudflare DNS API token).
- Planned end state after NetBird cutover (N3): cloudflared carries ONLY (1) the NetBird control plane `netbird.reverseproxyserver.net` (management+signal gRPC, relay WebSocket, embedded-Dex OIDC, dashboard) and (2) Nextcloud PUBLIC link-shares (the owner shares files with outsiders). Partners (2 people) reach Nextcloud + Jellyfin via Twingate. Admin services go NetBird-only.
- Limitations hit so far on the Cloudflare free tunnel: long-lived gRPC streams cut (~100 s) → NetBird signal "didn't receive a registration header" loop, and the primary NetBird client hangs; gRPC had to be enabled in the CF dashboard and Bot Fight Mode disabled for the NetBird client; the relay path through the tunnel stuttered Jellyfin earlier; QUIC relay dial fails (`tls: no application protocol`). Also check/report other known free-tier limits relevant to us (request body/upload size limit for Nextcloud, ToS on video streaming, WebSocket/idle timeouts).
- The home connection reportedly has a public IPv4 (66.164.11.87 noted 2026-09-17; not CGNAT). VERIFY without secrets: compare `curl -s https://ifconfig.me` (or api.ipify.org) from primary to that value, and note whether it may be dynamic (UNVERIFIED unless the ISP is known). The router/gateway is owner-managed (no access).
- Owner decisions to respect: UDP 3479 STUN stays CLOSED unless streaming stutters; minimal inbound exposure is valued; primary is RAM-constrained (anything new must be light); server-01 (192.168.1.90) is another on-LAN host; no VPS exists today — a paid VPS is a cost the owner must approve (a genuinely free tier, e.g. Oracle Cloud Always Free, may be listed but flag reliability/ToS/account risk).
- DNS for reverseproxyserver.net is on Cloudflare. Keeping Cloudflare as a DNS-only (grey-cloud) provider is NOT the same as using the tunnel — say clearly whether DNS can stay there for free (and whether DNS-01 certs need a CF API token), vs. moving DNS elsewhere.
TASK
1. Part A — current use: from the inventory + a read-only look at the cloudflared container and Traefik (labels on running containers mentioning reverseproxyserver.net hosts), list what cloudflared does today (hostnames, TLS termination, hidden home IP, DDoS/WAF, no inbound ports, http-01 cert path), and what it will still do post-NetBird. Identify every FUNCTION a replacement must cover (e.g. public reachability of the NetBird control plane incl. HTTP/2 gRPC long-lived streams + WebSocket, public Nextcloud shares incl. large uploads, TLS certs, IP hiding (want or not), DDoS/abuse protection, no/min open ports).
2. Part B — alternatives (free + open source), at minimum: (i) NO tunnel: port-forward TCP 443 to Traefik with DNS-01 wildcard certs + CrowdSec (bouncer for Traefik) + rate limits/geo-blocking + dynamic DNS if needed; (ii) Pangolin (fosrl/pangolin, self-hosted tunneled reverse proxy w/ WireGuard "Newt"); (iii) frp / rathole / other self-hosted tunnels; (iv) NetBird's own built-in reverse proxy feature (NetBird v0.65+ "Built-in Reverse Proxy with Custom Domains", https://netbird.io/knowledge-hub/reverse-proxy — what it does, whether self-hosted supports it, whether it could publish Nextcloud shares); (v) zrok/OpenZiti or others you find credible. For each: does it need a VPS (cost), inbound ports on the home router, gRPC long-stream + WebSocket support, TLS handling, IP hiding, DDoS/abuse posture, RAM footprint, maturity/maintenance/licence, fit for NetBird control plane AND for Nextcloud public shares.
3. Security analysis of the leading option(s) vs today: what new exposure is created (e.g. TCP 443 open to the internet on primary's Traefik), what mitigations (CrowdSec, Authelia, NetBird-only for admin hosts so only 2 hostnames are public, Traefik router allowlists), and residual risk. Be concrete.
4. Recommendation: ONE path to fully retire cloudflared (+ an alternative if it needs money), with a phased migration plan that keeps everything working (e.g. P1 DNS-01 wildcard certs; P2 CrowdSec; P3 move netbird host off tunnel; P4 Nextcloud shares; P5 remove remaining tunnel routes with NetBird cutover; P6 delete cloudflared), each with a verification and rollback. Note interactions with NetBird N2/N3 (the signal problem should disappear once the control plane is off the tunnel — explain why), Twingate (partners), and whether UDP 3479 decision changes.
5. Write /opt/appdata/docker/research/cloudflared_replacement.md: Verdict up top (plain language) / Part A table / required functions / alternatives comparison table / security analysis / recommendation + phased plan / risks / owner questions (only real decisions) / Sources list.
SCOPE ALLOWLIST: create /opt/appdata/docker/research/cloudflared_replacement.md; the context.md append below. Nothing else.
PERSIST BEFORE YOU FINISH
- Append (one `cat >>` heredoc; append only) a dated block "## cloudflared replacement research — 2026-09-25 (background agent)" to "/opt/appdata/docker/Machines/infrastructure general questions/.claude/context.md": What was done / Recommendation / Next step.
- Run: python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5
- FINAL message = wrap-up JSON only, always:
{"status":"succeeded|partially_succeeded|failed","project":"cloudflared-retirement (NetBird #258)","actions_taken":[],"actions_failed":[],"files_touched":[],"containers_restarted":[],"unverified_claims":[],"current_functions":[],"post_netbird_functions":[],"public_ip_verified":null,"alternatives":[{"name":"","needs_vps":null,"inbound_ports":"","grpc_streams_ok":null,"ip_hidden":null,"ram":"","verdict":""}],"recommendation":"","phases":[],"new_exposure":"","owner_questions":[],"report":"","next_step":"","notes":""}
claude-config/config/prompts/media-pipeline/AV-CHECK_detector_accuracy.md
+56
View File
@@ -0,0 +1,56 @@
# AV-CHECK detector accuracy — exact prompt as spawned (2026-09-24)
Tracks: personal_projects 61. Recovered verbatim from the session transcript 2026-09-25 (it was spawned inline and not saved at the time).
---
You are a bounded background agent for the media_pipeline project. Budget: --max-turns 25.
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up JSON with
status=partially_succeeded.
HARD RULES
- READ-ONLY investigation. NEVER restart, stop, recreate, or exec a mutating command in ANY existing container. Do not touch
the server-01 sandbox (another agent is using it) or ntfy (another agent is fixing it).
- Throwaway test containers ONLY on PRIMARY (this host): `docker run --rm --memory 4g --cpus 4`; check `free -m` before
each start and do not start one if MemAvailable < 6 GB (primary is RAM-constrained; OOM-kills have happened). Media is
mounted read-only at /m (NEVER at /lib). Write test files only inside the container's /tmp. Confirm none left running.
- NO git add/commit/push. NO DB writes (read-only SELECTs, one statement per psql -c, \d before guessing columns). No
notifications. Mask secrets. No sudo. Blocked → partial wrap-up, never work around.
- Do not change code. Report findings + ONE recommendation.
PERSIST: append a dated "## AV-CHECK — <date> (background agent)" block (What was done / Findings / State / Next) to
/opt/appdata/docker/docker-compose/media-downloader-local/.claude/context.md (single append); run
python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5.
FINAL message = wrap-up JSON only: status, project ("media_pipeline"), actions_taken[], actions_failed[], files_touched[],
containers_restarted[] (must be empty), unverified_claims[], next_step, notes + extra fields.
==========================================================================================
PHASE AV-CHECK — does the LIVE A/V-sync detector (and diagnose_av_sync.py) recover known offsets at FULL resolution?
Write ONE new file: /opt/appdata/docker/docker-compose/media-transcoder/design/INVESTIGATION_av_detector_accuracy.md.
Why: MP-11A (addendum DESIGN_sync_feedback_capture_ADDENDUM_offset_method.md — read its test section) found
diagnose_av_sync.py never recovered a known shift, but on 320-px re-encoded 120 s excerpts, which may cripple a
motion-onset method. The live detector in /opt/appdata/docker/docker-compose/media-transcoder/media-transcoder.py
(A/V-sync Phase 2 "MP-8" live since 2026-09-23; find its detection entry point — grep av_offset / detect / onset / RMS —
and the Stage-1 container-offset step with AV_CONTAINER_THRESHOLD_MS=150) shares that algorithm and decides every episode's
correction. We need to know whether production detection works at full resolution.
Method (all inside throwaway containers from image gitea.local/backtalk6858/media-transcoder:latest, which has the script's
Python deps + ffmpeg; mount the live script read-only too):
1. Pick 4 source files at FULL resolution (no video re-encode): 2 English-dubbed anime, 1 subbed anime, 1 live action, from
"/media/mediashare/MY MEDIA/MY MEDIA/seedbox local/webdav/Finished/Qbittorrent/" (e.g. the Yameii Frieren/Slime/Jobless
dubs, Farming Life, The Pitt S02E13). Cut a representative excerpt with STREAM COPY only (-c copy; same length the
detector normally analyzes — read the code for its window/segment size), then make shifted copies with -itsoffset on the
audio input + -c copy: 0, +150, +300, −300, +700 ms.
2. Run the detector's own function(s) on each (import media-transcoder.py as a module inside the container, or call the
minimal code path — do not modify the file; if import side effects make that impossible, copy the needed functions into a
container-/tmp harness verbatim and say so). Also run diagnose_av_sync.py
(/opt/appdata/docker/non-docker-python-scripts/diagnose_av_sync.py) on the same files. Record recovered offset, confidence,
method path taken (Stage-1 container offset vs content detection), runtime.
3. Separate the two mechanisms: -itsoffset shifts the CONTAINER timestamps, which Stage-1 may catch by itself. So ALSO make
one content-level shift that leaves container start times equal (e.g. atrim/adelay-based re-encode of AUDIO ONLY with
the start pts reset, video stream-copied) for +300 and −300 ms — that is the real "audibly out of sync" case (like
Slime S04E12, job 2702). Report both.
4. Interpret: accuracy table per file × shift × method; does content detection work at full res; was MP-11A's failure an
excerpt-resolution artifact; implications for MP-8 (learned-skip trusts clean detections) and MP-11 (auto_agreement rate).
Extra wrap-up fields: {"report":"","results":[{"file":"","shift_ms":0,"kind":"container|content","detector_ms":null,
"diagnose_ms":null,"stage":"","runtime_s":null}],"detector_verdict":"works|partial|broken","mp11a_artifact":true,
"recommendation":""}
claude-config/config/prompts/media-pipeline/AV-FIX_stop_stage2.md
+86
View File
@@ -0,0 +1,86 @@
# AV-FIX stop stage2 — exact prompt as spawned (2026-09-24)
Tracks: personal_projects 61. Recovered verbatim from the session transcript 2026-09-25 (it was spawned inline and not saved at the time).
---
You are a bounded background agent for the media_pipeline project. Budget: --max-turns 30.
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with
status=partially_succeeded. (Exception: a bounded WAIT — one ssh command that loops remotely with
`timeout <=900 bash -c 'until <cond>; do sleep 30; done'` — may be repeated at most 3 times per test step.)
HARD RULES
- NEVER restart, stop, recreate, or exec a mutating command in ANY live/prod container, on any host. Read-only
`docker exec ... sha256sum|ffprobe|cat` and `docker ps/inspect` on live containers are allowed.
- Container restart allowlist starts EMPTY. Before you start/stop/restart a SANDBOX container, add it to your allowlist and
log "ELEVATED <name>" in actions_taken; verify healthy after; list it in containers_restarted. Only
`media-pipeline-sandbox-*` containers on server-01 may ever be elevated.
- NO git add/commit/push. NO writes to the primary host's Postgres (read-only SELECTs only).
- Issue every mutating command as its OWN tool call; verify with a SEPARATE call.
- Mask secrets. No sudo. If a security hook/permission BLOCKS an action, do not work around it: emit a partial wrap-up.
- Do not make design decisions — they are pre-made below; if reality contradicts them, STOP and report.
ACCESS MAP
- Prod DB (READ-ONLY): PG=$(docker ps --format '{{.Names}}' | grep '^postgres-'); docker exec "$PG" psql -U postgres
-d media_pipeline -c "<one SELECT>"
- server-01: ssh administrator@192.168.1.90. Sandbox dir (server-01 + git twin on primary at the same path):
/opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/ (read README.md). Containers (keep healthy):
media-pipeline-sandbox-{postgres,media-transcoder-sandbox,media-api-sandbox,media-downloader-local-sandbox}-1;
filebot-monitor-sandbox stays OFF. The sandbox transcoder runs VERIFY_STREAMS_MODE=shadow — leave it.
- Baseline reset: TRUNCATE pipeline_learning, transcode_jobs, transcode_overrides, pipeline_events, download_jobs,
rename_jobs, show_configs RESTART IDENTITY; pipe init/02-seed.sql
(docker exec -i media-pipeline-sandbox-postgres-1 psql -U sandbox -d media_pipeline_sandbox < file); then restart
media-transcoder-sandbox AND media-downloader-local-sandbox (startup caches).
- ROOT-owned outputs: clear via docker exec media-pipeline-sandbox-media-transcoder-sandbox-1 find <dir> -mindepth 1 -delete.
- Fixtures: server-01 <sandbox>/sandbox-fixtures/mp10/ ("MP10 D1 - S01E01.mkv", "MP10 L1 offset - S01E02.mkv" (+300 ms
container shift), "MP10 M1 offset (1993).mkv" (−250 ms), "MP10 S2 - S02E02.mkv") and mp8/. Stage by cp into
sandbox-input/<CATEGORY DIR>/<Show>/ (D → "H264 ENGLISH DUBBED ANIME TO BE TRANSCODED"/"MP10 D", S → "H264 ENGLISH SUBBED
ANIME TO BE TRANSCODED"/"MP10 S", L → "H264 LIVE ACTION SERIES TO BE TRANSCODED"/"MP10 L", M → "H264 MOVIES TO BE
TRANSCODED"/"MP10 M1 (1993)"). ≤3 staged at a time; sandbox-input EMPTY at the end.
- Script workflow: cp the LIVE /opt/appdata/docker/docker-compose/media-transcoder/media-transcoder.py to the primary
sandbox dir as the working copy (overwrite the old one there); BASELINE sha256 must start 31012734a3da2e57 (MP-10, promoted
20:15 today) — if not, STOP. Edit; py_compile; scp to server-01; restart the sandbox transcoder; confirm the container's
/app/media-transcoder.py sha256 equals the working copy.
- Promotion (only on GO): if live sha256 still equals BASELINE, cp working copy → live path; verify host hash AND
`docker exec media-transcoder-media_transcoder-1 sha256sum /app/media-transcoder.py`. Do NOT restart live; output the
human restart command (cd /opt/appdata/docker/docker-compose/media-transcoder && docker compose restart media_transcoder).
PERSIST: append "## AV-FIX (stage-2 suppression) — <date> (background agent)" to
/opt/appdata/docker/docker-compose/media-downloader-local/.claude/context.md (What/Decisions/State/Next); run
python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5. FINAL message = wrap-up JSON only:
status, project ("media_pipeline"), actions_taken[], actions_failed[], files_touched[], containers_restarted[],
unverified_claims[], next_step, notes + extra fields.
==========================================================================================
PHASE AV-FIX — STOP APPLYING the Stage-2 (motion-onset) A/V correction; keep Stage 1. ONE feature. Sandbox → promote.
Evidence (read /opt/appdata/docker/docker-compose/media-transcoder/design/INVESTIGATION_av_detector_accuracy.md): Stage 1
(_detect_container_av_offset ~L558, AV_CONTAINER_THRESHOLD_MS=150 at L177) measured container shifts exactly 16/16; Stage 2
(_analyze_av_window ~L429, used by detect_av_sync_offset ~L599) recovered 0/24 known shifts and produces 1–3 s false lags.
Prod: 5 jobs got Stage-2-applied corrections (e.g. Slime S02E12 +2120 ms, which the owner heard out of sync).
PRE-DECIDED DESIGN:
1. New module constant AV_STAGE2_APPLY = os.environ.get("AV_STAGE2_APPLY", "0") == "1" (default OFF; comment why, cite the
investigation doc). When Stage 1 does not produce a correction and Stage 2 runs and returns a nonzero offset, and
AV_STAGE2_APPLY is off: do NOT apply it — the job proceeds with offset 0 exactly as if Stage 2 had found nothing; log one
INFO line "stage-2 suggested <ms> ms (conf <c>) — NOT applied (AV_STAGE2_APPLY=0)"; and post a pipeline event
`av_sync.stage2_suppressed` with {suggested_ms, confidence, windows} through the SAME event path the script already uses
for other events (find it; do not invent a new transport). The transcode_jobs av_offset_ms written = 0 (what was
applied); keep detection_method semantics unchanged.
2. Stage 1 behavior, the MP-8 learned-skip gate/canary, and the pipeline_learning av_sync rows are UNCHANGED in this fix
(learned-skip now only skips an inert stage; its statistic is revisited when MP-11 replaces Stage 2 with the
subtitle-vs-VAD aligner — note this in context.md, do not change it).
3. diagnose_av_sync.py is NOT touched (its sign inversion is fixed in the MP-11 build).
TESTS:
a. Unit (stdlib unittest, new file test_av_stage2_suppression.py next to the working copy): monkeypatch Stage 1 to return
no correction and Stage 2 to return a confident nonzero (e.g. +2120) → returned/applied offset 0 with suppression logged;
with AV_STAGE2_APPLY=1 → +2120 applied (old behavior preserved behind the flag); Stage 1 nonzero → still applied
regardless of the flag. Also rerun the existing test_verify_output_streams.py (must stay 36/36).
b. Sandbox E2E regression (baseline reset between batches): "MP10 L1 offset" → Stage 1 still corrects ≈ −306 ms;
"MP10 M1 offset" → ≈ +213 ms corrected; "MP10 D1" and "MP10 S2" → done, av_offset_ms 0, MP-10 shadow verdict rows still
written. Grep the sandbox logs for any "stage-2 suggested" lines and quote them (if Stage 2 happened to suggest
something on a clean fixture, that is a live demonstration of the fix).
GO = all unit tests pass AND all 4 E2E rows as expected → PROMOTE. Else NO-GO, no promotion.
Clean up: sandbox-input empty, outputs cleared, DB baseline reset, all 4 sandbox containers healthy.
Extra wrap-up fields: {"baseline_sha256":"","working_sha256":"","unit_tests":"","e2e":[{"fixture":"","expected":"",
"actual":"","pass":true}],"stage2_suggestions_seen":[],"verdict":"GO|NO-GO","promoted":false,"live_sha256_after":"",
"human_restart_cmd":""}
claude-config/config/prompts/media-pipeline/MP-10_build_verification_gate.md
+115
View File
@@ -0,0 +1,115 @@
# MP-10 build verification gate — exact prompt as spawned (2026-09-24)
Tracks: personal_projects 61. Recovered verbatim from the session transcript 2026-09-25 (it was spawned inline and not saved at the time).
---
You are a bounded background agent for the media_pipeline project. Budget: --max-turns 45 (raised: code build + 11-fixture sandbox matrix).
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with
status=partially_succeeded. (Exception: a bounded WAIT — one ssh command that loops remotely with
`timeout <=900 bash -c 'until <cond>; do sleep 30; done'` — may be repeated at most 3 times per test step.)
HARD RULES
- NEVER restart, stop, recreate, or exec a mutating command in ANY live/prod container, on any host.
Read-only `docker exec ... sha256sum|ffprobe|cat` and `docker ps/inspect` on live containers are allowed.
- Container restart allowlist starts EMPTY. Before you start/stop/restart a SANDBOX container, add it to your allowlist
and log "ELEVATED <name>" in actions_taken; verify it is healthy after, and list it in containers_restarted. Only
`media-pipeline-sandbox-*` containers on server-01 may ever be elevated.
- NO git add/commit/push. NO writes to the primary host's Postgres (read-only SELECTs only). The main session does all
DB bookkeeping.
- Issue every mutating command as its OWN tool call; verify with a SEPARATE call (never mutate+sleep+verify in one sh -c).
- Mask secrets in all output. No sudo. If a step truly needs a privilege you lack, end your run with a final message headed
exactly `⏸ OWNER-COMMAND-REQUEST` containing host:, command:, why:, paste_back:, resume_at: — then stop (max 3 per run).
If a security hook/permission BLOCKS an action, do not work around it: emit a partial wrap-up.
- Do not make design decisions. The spec is the design doc + owner decisions below; if reality contradicts them (code anchor
missing, fixture absent), STOP and report.
- Two other agents run in parallel (a read-only detector check on primary; an ntfy fix). Do not touch ntfy, and do not run
heavy containers on PRIMARY (all your transcodes happen in the server-01 sandbox).
ACCESS MAP
- Prod DB (READ-ONLY): PG=$(docker ps --format '{{.Names}}' | grep '^postgres-'); docker exec "$PG" psql -U postgres
-d media_pipeline -c "<one SELECT>" (one statement per -c)
- server-01: ssh administrator@192.168.1.90 (batch several read commands per ssh call).
- Sandbox dir (server-01, and a git-tracked twin on the primary host at the same path):
/opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/ (read its README.md first)
- Sandbox containers (all must stay healthy): media-pipeline-sandbox-{postgres,media-transcoder-sandbox,media-api-sandbox,
media-downloader-local-sandbox}-1. filebot-monitor-sandbox is deliberately OFF (compose profile mp14) — leave it off.
DB media_pipeline_sandbox, user sandbox (no password via docker exec). Pipe SQL files with
docker exec -i media-pipeline-sandbox-postgres-1 psql -U sandbox -d media_pipeline_sandbox < <file>
- Sandbox baseline reset: TRUNCATE pipeline_learning, transcode_jobs, transcode_overrides, pipeline_events, download_jobs,
rename_jobs, show_configs RESTART IDENTITY; then pipe init/02-seed.sql; then restart media-transcoder-sandbox AND
media-downloader-local-sandbox (both cache state at startup — a file already done in the old DB is silently skipped).
- sandbox-output/renamed/tmp hold ROOT-owned files: clear from inside the container
(docker exec media-pipeline-sandbox-media-transcoder-sandbox-1 find <dir> -mindepth 1 -delete), not host rm.
- Fixtures (already cut, verified 2026-09-24): server-01 <sandbox>/sandbox-fixtures/mp10/ :
"MP10 D1 - S01E01.mkv" … "MP10 D5 - S01E05.mkv", "MP10 S1 - S02E01.mkv", "MP10 S2 - S02E02.mkv", "MP10 S3 - S02E03.mkv",
"MP10 L1 - S01E01.mkv", "MP10 L1 offset - S01E02.mkv" (+300 ms audio late), "MP10 M1 (1993).mkv",
"MP10 M1 offset (1993).mkv" (−250 ms). Layouts exactly as the design §6 table (D2 TrueHD start fixed to 0.00).
STAGE = cp a clip into sandbox-input/<CATEGORY DIR>/<Show>/ with category dirs "H264 ENGLISH DUBBED ANIME TO BE
TRANSCODED" (D*), "H264 ENGLISH SUBBED ANIME TO BE TRANSCODED" (S*), "H264 LIVE ACTION SERIES TO BE TRANSCODED" (L*),
"H264 MOVIES TO BE TRANSCODED" (M*); use show dir "MP10 D" / "MP10 S" / "MP10 L" and movie dir "MP10 M1 (1993)".
sandbox-input/ must be EMPTY when you finish; clear sandbox-output/renamed/tmp between runs.
- The sandbox encodes with software libx265 (server-01 GPU is NVIDIA — no VAAPI). Expected. Stage at most 3 clips at a
time (1 worker; keeps runtime bounded).
- Script edit workflow: cp the LIVE /opt/appdata/docker/docker-compose/media-transcoder/media-transcoder.py to the
primary-host sandbox dir as your working copy; BASELINE sha256 must be 6188a6a59dd90aae… (full value: sha256sum the live
file at start and record it; if its first 16 hex != 6188a6a59dd90aae STOP). Edit the working copy; python3 -m py_compile;
scp to the same path on server-01; restart the sandbox transcoder; confirm `docker exec
media-pipeline-sandbox-media-transcoder-sandbox-1 sha256sum /app/media-transcoder.py` equals the working copy's sha256.
- Promotion (only when verdict GO): if the live file's sha256 still equals BASELINE, `cp <working copy> <live path>`
(cp keeps the inode); verify the host hash AND `docker exec media-transcoder-media_transcoder-1 sha256sum
/app/media-transcoder.py` both equal the working copy. Do NOT restart live — put the human's restart command in your output
(cd /opt/appdata/docker/docker-compose/media-transcoder && docker compose restart media_transcoder).
- Logs: ssh ... "docker logs --since <ts> media-pipeline-sandbox-media-transcoder-sandbox-1 2>&1 | grep -E '<pattern>'"
PERSIST BEFORE YOU FINISH
- Append a dated block "## MP-10 BUILD — <date> (background agent)" to
/opt/appdata/docker/docker-compose/media-downloader-local/.claude/context.md (What was done / Decisions / Current state /
Next step — enough for a fresh agent to resume). Single append (other agents append too).
- Run: python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5
- FINAL message = wrap-up JSON only (success or failure): status, project ("media_pipeline"), actions_taken[],
actions_failed[], files_touched[], containers_restarted[], unverified_claims[], next_step, notes + extra fields below.
==========================================================================================
PHASE MP-10 BUILD — implement `verify_output_streams()` per
/opt/appdata/docker/docker-compose/media-transcoder/design/DESIGN_verification_hardening.md (READ IT FULLY — §4 checks and
decision tables, §5 failure handling, §6 fixture matrix, §7 owner banner). SANDBOX → PROMOTE in SHADOW mode.
OWNER DECISIONS (fixed): all 8 §7 recommendations confirmed — shadow-first; quarantine under <tmp_root> pruned by
FAILED_JOB_MAX_AGE_DAYS; forced full-dialogue only-track = WARN (not fail); failed_verification is a HELD state not counted
toward MAX_RETRY_FAILURES; add `category=` to output_verification AND to existing audio_selection/subtitle_selection learning
keys (new rows only); density thresholds 0.35×/0.6×/4-per-min as module constants; no Whisper. The forced-sub PICKER fix is
NOT part of this build (it is MP-15) — do not change _find_forced_subtitle_idx selection logic.
Owner hard rule 2026-09-24: EVERY English-dubbed output must carry a forced subtitle track with the default flag (episode
titles, signs, foreign-language scenes) → `dub_signs_missing` stays a FAIL.
Facts: prod transcode_jobs.status has NO CHECK constraint (no DDL needed). Code anchors in the live script:
verify_output_playback() ~L2433, transcode_file_chunked() ~L2484, get_done_paths() ~L3074, _api_job_status() ~L3382,
NtfyNotifier ~L3242, _find_forced_subtitle_idx ~L1347, MAX_RETRY_FAILURES L151. media-api's own retranscode acceptance of
failed_verification is OUT of scope (MP-13) — note it, don't touch media-api.
Steps
1. Implement (working copy only): pure verify_output_streams(src_streams, out_streams, category, selection) returning a
verdict {status: pass|warn|fail, codes[], checks[]}; VERIFY_STREAMS_MODE env (off|shadow|enforce, default "shadow");
placement after verify_output_playback(); §5 failure path (status failed_verification, source kept, output quarantined,
ntfy via NtfyNotifier, one learning row per check with category=) active only in enforce; in shadow: log + learning rows
only, job proceeds exactly as today. Add failed_verification to the skip sets in get_done_paths() and _api_job_status().
Keep functions short; comments explain why.
2. Pure-function unit tests (a new file next to the working copy: test_verify_output_streams.py, stdlib unittest): saved
ffprobe-shaped dicts for every row of §4.3/§4.4 incl. the Fire Force layout (a:0 truehd eng, a:1 aac untagged, mapped 0)
and X1 (duration 50%). Run them; all pass.
3. Sandbox E2E, VERIFY_STREAMS_MODE=enforce set ONLY for the sandbox transcoder service (edit the primary-twin compose env and
scp it; note it in files_touched) — enforce is needed to exercise §5. Run the §6 matrix in batches of ≤3 (baseline reset +
restart between batches). For every fixture record: expected vs actual verdict + codes, job status, quarantine location,
deletion_eligible_after (must be NULL for failed_verification), learning rows (quote), ntfy attempted (sandbox NTFY_URL is
empty → confirm the code path ran without error). L1-offset ≈ +300 ms and M1-offset ≈ −250 ms: quote detected offsets.
Pass criteria = design §6 (every row gets its expected verdict; D1, S2, M1 zero fails; quarantined outputs never in
sandbox-renamed).
4. Then set the sandbox transcoder back to VERIFY_STREAMS_MODE=shadow, rerun D2 only: job must finish `done` exactly like
today with a shadow `fail` learning row → proves shadow is inert.
5. Verdict GO only if 2–4 all pass. GO → PROMOTE (default mode shadow; do NOT add VERIFY_STREAMS_MODE to the live compose).
NO-GO → no promotion; report the failing rows.
6. Clean up: sandbox-input empty, outputs/quarantine cleared, DB baseline reset, transcoder-sandbox left on shadow and
healthy, all 4 sandbox containers healthy.
Extra wrap-up fields: {"baseline_sha256":"","working_sha256":"","unit_tests":"N/N PASS","matrix":[{"fixture":"D1",
"expected":"","actual":"","codes":[],"pass":true}],"shadow_inert":"PASS|FAIL","verdict":"GO|NO-GO","promoted":false,
"live_sha256_after":"","human_restart_cmd":""}
claude-config/config/prompts/media-pipeline/MP-15_forced_sub_picker.md
+67
View File
@@ -0,0 +1,67 @@
# MP-15 — forced-subtitle PICKER fix (category-aware) — sandbox build, verdict only
Tracks: personal_projects 61 (fix inventory row 9). Owner decisions 2026-09-25 (see below). Written + spawned 2026-09-25.
> **⛔ SUPERSEDED 2026-09-25 18:05 (NOT spawned).** Owner redirected MP-15 to a CONTENT-FIRST picker (subtitle
> density decides signs vs dialogue; keywords only break ties). Thresholds come from the calibration scan
> `MP-15a_subtitle_density_scan.md`. This keyword-split version is kept for history; MP-15 v2 will be saved as
> `MP-15_v2_density_picker.md` after the scan.
---
You are a bounded background agent for the media_pipeline project. PHASE MP-15 — forced-subtitle picker fix in the SANDBOX, verify, deliver a verdict. Budget: --max-turns 40 (reason: function change + unit tests + a derived fixture + 7 short sandbox runs with an old-code baseline).
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with status=partially_succeeded.
HARD RULES
- NEVER restart, stop, recreate, or exec a mutating command in ANY live/prod container, on any host. Read-only `docker exec ... sha256sum|ffprobe|cat` and `docker ps/inspect` on live containers are allowed.
- Container restart allowlist starts EMPTY. Before you start/stop/restart a SANDBOX container, add it to your allowlist and log "ELEVATED <name>" in actions_taken; verify healthy after; list it in containers_restarted. Only `media-pipeline-sandbox-*` containers on server-01 may be elevated. Throwaway `docker run --rm` containers you create are fine.
- NO git add/commit/push. NO writes to the primary host's Postgres. NO edits to the LIVE media-transcoder.py. NO promotion — verdict only; the main session promotes.
- Issue every mutating command as its OWN tool call; verify with a SEPARATE call.
- WAITING: never hand-write nested-quote wait loops. Write wait conditions as small script FILES on the target host (SQL in a .sql file run with `psql -f`), run with `timeout 900`. Before wrap-up confirm no leftover waiters on both hosts (`ps -eo pid,args | grep "[t]imeout"`; kill only your own).
- Mask secrets. No sudo. If a security hook BLOCKS an action, do not work around it: partial wrap-up.
- Do not make design decisions beyond the pre-made ones below. If reality contradicts them, STOP and report.
ACCESS MAP
- server-01: ssh administrator@192.168.1.90 (batch read commands per ssh call).
- Sandbox dir (server-01, and a git-tracked twin on the primary host at the same path): /opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/
- Sandbox containers: media-pipeline-sandbox-media-transcoder-sandbox-1, media-pipeline-sandbox-postgres-1 (DB media_pipeline_sandbox, user sandbox, no password via docker exec).
- Baseline reset: `TRUNCATE pipeline_learning, transcode_jobs, transcode_overrides, pipeline_events, download_jobs, rename_jobs, show_configs RESTART IDENTITY;` then `docker exec -i media-pipeline-sandbox-postgres-1 psql -U sandbox -d media_pipeline_sandbox < init/02-seed.sql`; ALWAYS restart the sandbox transcoder after a reset (done-set cached at startup).
- sandbox-output/, sandbox-renamed/, sandbox-tmp/ hold ROOT-owned files: clear them from inside the container (`docker exec media-pipeline-sandbox-media-transcoder-sandbox-1 find <in-container path> -mindepth 1 -delete`; get paths via docker inspect).
- STAGE = cp a fixture into sandbox-input/<CATEGORY DIR>/<Show>/ — category dirs: "H264 ENGLISH DUBBED ANIME TO BE TRANSCODED", "H264 ENGLISH SUBBED ANIME TO BE TRANSCODED", "H264 LIVE ACTION SERIES TO BE TRANSCODED", "H264 MOVIES TO BE TRANSCODED" (path drives preset + category — never flatten; use Show dirs named after the fixture, e.g. "MP10 D3", and for movies stage the file directly in the category dir or in a "MP10 M1 (1993)" folder — check how verify_category() and get_preset_path_for_input() read the path). sandbox-input/ must hold only the 4 empty category dirs at the end.
- Sandbox = software libx265 (server-01 GPU is NVIDIA, no VAAPI). Expected; this fix is encoder-independent.
- Script workflow: the primary-host working copy /opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/media-transcoder.py must equal live (sha 7617fc0a…, record the live sha256 as BASELINE and confirm; the sandbox container runs it too). Edit the working copy; `python3 -m py_compile`; scp to the same path on server-01; restart the sandbox transcoder; confirm the in-container sha256 matches.
- Logs: `ssh administrator@192.168.1.90 "docker logs --since <ts> media-pipeline-sandbox-media-transcoder-sandbox-1 2>&1 | grep -E '<pattern>'"`. The picker logs "Forced subtitle (...)" / "Only one English subtitle track" lines and MP-10 logs "[VERIFY]" lines.
- Unit tests next to the working copy on primary: test_vaapi_cmd.py, test_verify_output_streams.py, test_av_stage2_suppression.py, test_mp19b_resume.py (82 total today). All must still pass, plus your new file.
BACKGROUND (diagnosed by the main session — do not re-diagnose)
- `_find_forced_subtitle_idx(streams)` (~L1436; single call site ~L1710 in the encode-plan builder) has 3 tiers: (1) disposition.forced=1; (2) title keyword from FORCED_SUBTITLE_TITLE_KEYWORDS (~L1400) — the loop is TRACK-outer / keyword-inner, so the FIRST TRACK matching ANY keyword wins: a dub with s:0 "English SDH" and s:1 "Signs & Songs" gets SDH forced (bug); (3) exactly one English/untagged track → force it. The keyword list includes dialogue-type labels ("hearing impaired", "closed captions", "sdh", "hi", "cc", "text") matched as SUBSTRINGS, so "hi" hits "Chinese"/"Hindi", "cc" hits "accessibility", "text" hits "context".
- MP-10 (DESIGN_verification_hardening.md §7 Q3) found full-dialogue tracks forced over English audio (sdh 65 rows, single_track 49 rows).
PRE-MADE DECISIONS (binding; owner-approved 2026-09-25)
- D1 Signature: `_find_forced_subtitle_idx(streams, category)`; the call site passes `verify_category(input_path)` (the existing helper ~L2816 returning "dubbed" / "subbed" / "live_action" / "movies" — confirm the exact strings and that input_path is in scope at the call site; if not, thread it through minimally).
- D2 Split the keyword list into SIGNS keywords (everything except dialogue-type) and DIALOGUE keywords = ["hearing impaired", "closed captions", "sdh", "hi", "cc", "text"]. DIALOGUE single words (sdh, hi, cc, text) match on WORD BOUNDARIES (regex \b…\b, case-insensitive); multi-word phrases keep substring matching. SIGNS keyword matching is unchanged (substring, same priority order).
- D3 Tier 1 (disposition.forced) unchanged for every category.
- D4 Tier 2a (every category): scan ALL English/untagged tracks for SIGNS keywords first → first track (in stream order) that matches any SIGNS keyword wins (keep multi-word-before-single-word keyword priority within a track as today).
- D5 Tier 2b DIALOGUE keywords: DUBBED → allowed as a fallback after 2a (OWNER RULE: every English-dub output keeps a default+forced sub for episode titles/foreign scenes; a full-dialogue track is better than none) — reason string `dialogue_fallback=<kw>`. SUBBED → behaviour must equal today's (subbed uses the dialogue track as the default sub; keep the old single-pass semantics for subbed exactly — regression guard). LIVE_ACTION / MOVIES → DIALOGUE keywords are IGNORED (never forced).
- D6 Tier 3 single_track: DUBBED and SUBBED → unchanged (force the only English track). LIVE_ACTION / MOVIES → do NOT force; return (None, "no_signs_track_live_movie") so the preset behaviour applies.
- D7 Log lines keep their existing wording for unchanged paths; new paths log at INFO with the reason string. Localized edits only; update the function docstring to describe the category rules and the owner rule.
FIXTURES (server-01 sandbox-fixtures/mp10/, cut earlier): "MP10 D3 - S01E03.mkv" (DUBBED: s:0 full "English", s:1 "Signs & Songs"), "MP10 D4 - S01E04.mkv" (DUBBED: s:0 "English", s:1 "English SDH" — same full track retitled, no signs), "MP10 L1 - S01E01.mkv" (LIVE: one sub "English SDH"), "MP10 M1 (1993).mkv" (MOVIES: s:0 full "English", s:1 "English Forced" disposition.forced=1), "MP10 S1 - S02E01.mkv", "MP10 S2 - S02E02.mkv", "MP10 S3 - S02E03.mkv" (SUBBED set). Read the fixture layouts with ffprobe before using them and report any mismatch with these descriptions.
- DERIVED FIXTURE you create (allowed): `sandbox-fixtures/mp15/MP15 D6 - S01E06.mkv` = D3 remuxed with s:0 retitled "English SDH" (so track order is SDH-titled full track first, "Signs & Songs" second), via a throwaway container on server-01: `ffmpeg -i "MP10 D3 - S01E03.mkv" -map 0 -c copy -metadata:s:s:0 title="English SDH" "MP15 D6 - S01E06.mkv"` (use the sandbox transcoder image with --entrypoint ffmpeg; mount at /w, NEVER /lib). ffprobe it to confirm.
STEPS
0 Baseline: BASELINE sha; working copy == sandbox container == live; sandbox-input empty; fixtures present; create D6; reset DB; restart transcoder (ELEVATE).
1 OLD-code evidence (no transcodes needed): run a tiny Python harness in a throwaway container (or on primary with the working copy imported via importlib) that ffprobes each fixture (`-show_streams -of json`) and calls the CURRENT `_find_forced_subtitle_idx(streams)`; record (idx, reason, title) per fixture. EXPECT D6 → SDH track (the bug), L1 → single_track/sdh forced, D4 → sdh.
2 Implement D1–D7; add `test_mp15_picker.py` (synthetic stream lists): dub SDH-first+signs → signs; dub only "English"+"English SDH" → sdh via dialogue_fallback; dub single untitled → single_track; live single "English SDH" → None; live "English"+"Signs" → signs; movie forced disposition → tier 1; subbed layouts → identical to old function (import the old function body copy inside the test as a reference, or compare against recorded step-1 results); word-boundary: titles "Chinese", "Hindi", "accessibility", "context" never match dialogue keywords; "English (CC)" and "English [SDH]" do. Run ALL unit test files — all pass. py_compile; scp; restart; confirm hash.
3 NEW-code harness: repeat step 1's harness with the new function + category for each fixture; EXPECT: D3 → s:1 signs (unchanged), D6 → s:1 signs (FIXED), D4 → s:1 sdh with reason dialogue_fallback=sdh (owner rule kept), L1 → None, M1 → s:1 tier 1 (unchanged), S1–S3 → identical to step 1.
4 Sandbox transcodes (end-to-end proof), one staged at a time or in category batches: D6 (dubbed), D4 (dubbed), L1 (live), M1 (movies), S2 (subbed). For each output ffprobe subtitle streams + dispositions and read the job's [VERIFY] lines. EXPECT: D6 output forced+default sub = the "Signs & Songs" track; D4 output forced+default = the SDH/full track (dialogue_fallback) and MP-10 verify = warn at most (never fail); L1 output has NO forced subtitle (per the live preset, possibly no subtitle stream at all) and MP-10 verify has no fail; M1 forced = "English Forced"; S2 unchanged vs expectations in DESIGN_verification_hardening.md §6 (pass subbed_dialogue_default). Un-stage and clear between batches; reset DB + restart between runs if a done-set would skip a file.
5 Verdict: GO iff steps 1–4 match every expectation and all unit tests pass. NO promotion. On NO-GO give the failing expectation with evidence.
6 Clean up: sandbox-input back to 4 empty dirs; clear sandbox-output/renamed/tmp; reset DB; sandbox transcoder RUNNING idle on the new working copy; keep sandbox-fixtures/mp15/D6 (regression fixture); no leftover waiters; delete your temp scripts by name.
SCOPE ALLOWLIST: primary working copy media-transcoder.py + the new test_mp15_picker.py next to it; server-01 sandbox dir contents (script copy, sandbox-input/output/renamed/tmp, sandbox-fixtures/mp15/) + sandbox DB; your temp scripts; the context.md append. Nothing else.
PERSIST BEFORE YOU FINISH
- Append (one `cat >>` heredoc; append only) "## MP-15 — 2026-09-25 (background agent)" to /opt/appdata/docker/docker-compose/media-downloader-local/.claude/context.md: What was done / Decisions / Current state / Next step.
- Run: python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5
- FINAL message = wrap-up JSON only, always:
{"status":"succeeded|partially_succeeded|failed","project":"media_pipeline","phase":"MP-15","actions_taken":[],"actions_failed":[],"files_touched":[],"containers_restarted":[],"unverified_claims":[],"baseline_sha256":"","new_sha256":"","old_code_picks":{},"new_code_picks":{},"transcode_results":{"D6":{},"D4":{},"L1":{},"M1":{},"S2":{}},"unit_tests":{"passed":null,"failed":null},"verdict":"GO|NO-GO","diff_summary":"","next_step":"","notes":""}
claude-config/config/prompts/media-pipeline/MP-15_v2_hybrid_picker.md
+73
View File
@@ -0,0 +1,73 @@
# MP-15 v2 — HYBRID forced-subtitle picker (labels first, density for untitled/ties) — sandbox build, verdict only
Tracks: personal_projects 61 (fix inventory row 9). Supersedes `MP-15_forced_sub_picker.md` (keyword split, never spawned).
Evidence: `/opt/appdata/docker/docker-compose/media-transcoder/design/CALIBRATION_subtitle_density.md` (+ .csv), MP-15a
scan 2026-09-25. Owner decisions 2026-09-25 (below). Written + spawned 2026-09-25.
---
You are a bounded background agent for the media_pipeline project. PHASE MP-15 v2 — hybrid forced-subtitle picker in the SANDBOX (+ the matching MP-10 guard), verify, deliver a verdict. Budget: --max-turns 45 (reason: picker rewrite + MP-10 guard + unit tests + old/new harness over 10 fixtures + 6 sandbox transcodes).
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with status=partially_succeeded.
HARD RULES
- NEVER restart, stop, recreate, or exec a mutating command in ANY live/prod container, on any host. Read-only `docker exec ... sha256sum|ffprobe|cat` and `docker ps/inspect` on live containers are allowed.
- Container restart allowlist starts EMPTY. Before you start/stop/restart a SANDBOX container, add it to your allowlist and log "ELEVATED <name>" in actions_taken; verify healthy after; list it in containers_restarted. Only `media-pipeline-sandbox-*` containers on server-01 may be elevated. Throwaway `docker run --rm` containers you create are fine.
- NO git add/commit/push. NO writes to the primary host's Postgres. NO edits to the LIVE media-transcoder.py. NO promotion — verdict only; the main session promotes.
- Issue every mutating command as its OWN tool call; verify with a SEPARATE call.
- WAITING: never hand-write nested-quote wait loops. Write wait conditions as small script FILES on the target host (SQL in a .sql file run with `psql -f`), run with `timeout 900`. Before wrap-up confirm no leftover waiters on both hosts (`ps -eo pid,args | grep "[t]imeout"`; kill only your own). Do NOT leave long-running background jobs and go idle — run each batch in the foreground with a timeout.
- Mask secrets. No sudo. If a security hook BLOCKS an action, do not work around it: partial wrap-up.
- Do not make design decisions beyond the pre-made ones below. If reality contradicts them, STOP and report.
ACCESS MAP
- server-01: ssh administrator@192.168.1.90 (batch read commands per ssh call).
- Sandbox dir (server-01, and a git-tracked twin on the primary host at the same path): /opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/
- Sandbox containers: media-pipeline-sandbox-media-transcoder-sandbox-1, media-pipeline-sandbox-postgres-1 (DB media_pipeline_sandbox, user sandbox, no password via docker exec).
- Baseline reset: `TRUNCATE pipeline_learning, transcode_jobs, transcode_overrides, pipeline_events, download_jobs, rename_jobs, show_configs RESTART IDENTITY;` then `docker exec -i media-pipeline-sandbox-postgres-1 psql -U sandbox -d media_pipeline_sandbox < init/02-seed.sql`; ALWAYS restart the sandbox transcoder after a reset.
- sandbox-output/, sandbox-renamed/, sandbox-tmp/ hold ROOT-owned files: clear them from inside the container (`docker exec media-pipeline-sandbox-media-transcoder-sandbox-1 find <in-container path> -mindepth 1 -delete`; paths via docker inspect).
- STAGE = cp a fixture into sandbox-input/<CATEGORY DIR>/<Show>/ — category dirs: "H264 ENGLISH DUBBED ANIME TO BE TRANSCODED", "H264 ENGLISH SUBBED ANIME TO BE TRANSCODED", "H264 LIVE ACTION SERIES TO BE TRANSCODED", "H264 MOVIES TO BE TRANSCODED" (path drives preset + category; read verify_category() and get_preset_path_for_input()). sandbox-input/ must hold only the 4 empty category dirs at the end.
- Sandbox = software libx265 (NVIDIA host, no VAAPI) — expected; this fix is encoder-independent.
- Script workflow: primary working copy /opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/media-transcoder.py must equal live (sha 7617fc0a…; record live sha256 as BASELINE; the sandbox container runs it). Edit; `python3 -m py_compile`; scp to server-01; restart the sandbox transcoder; confirm the in-container sha256.
- Logs: `ssh administrator@192.168.1.90 "docker logs --since <ts> media-pipeline-sandbox-media-transcoder-sandbox-1 2>&1 | grep -E '<pattern>'"`.
- Unit tests next to the working copy on primary: test_vaapi_cmd.py, test_verify_output_streams.py, test_av_stage2_suppression.py, test_mp19b_resume.py (82 today). All must still pass, plus your new test file.
BACKGROUND (main session + MP-15a scan — do not re-diagnose)
- Current `_find_forced_subtitle_idx(streams)` (~L1436, one call site ~L1710): tier 1 disposition.forced; tier 2 FORCED_SUBTITLE_TITLE_KEYWORDS (~L1400) with a TRACK-outer/keyword-inner loop (first track matching ANY keyword wins → a dub with s:0 "English SDH", s:1 "Signs & Songs" forces SDH); dialogue labels (sdh/hi/cc/text/hearing impaired/closed captions) are matched as SUBSTRINGS ("hi" hits "Chinese"); tier 3 forces a lone English track.
- Calibration (75 files / 129 streams): clean signs tracks ≤ 3.7 events/min, dialogue ≥ 10.1 — nothing between. BUT typeset-heavy BD fansub signs tracks are DENSE (100–15000 epm; e.g. Bookworm S01E27 "Signs & Songs [EMBER]" forced=1 at 300 epm vs Dialogue 315) → density must NOT override a signs label/flag. Untitled sparse signs exist (Yu Yu Hakusho 001: untitled track with 1 event). All 20 single-English-track files were dialogue. [Yameii] CR dubs carry "English Signs [Generated]" with forced=0 (label, not flag). Prod history: title_keyword=sdh 65, single_track 49.
- MP-10's `_SubProfile.signs_like/dialogue_like` (~L2978–2992, constants VERIFY_SIGNS_MAX_RATIO 0.35 / VERIFY_DIALOGUE_MIN_RATIO 0.6 / VERIFY_DIALOGUE_MIN_PER_MIN 4.0 ~L2801) will call typeset-heavy signs "dialogue-like" and raise false `full_subs_forced_only_track` on fansub dubs once MP-10 is enforced (~10-01).
PRE-MADE DECISIONS (binding; owner-approved 2026-09-25)
- D1 Signature `_find_forced_subtitle_idx(streams, category, sub_counts=None)`; the call site passes `verify_category(input_path)` and per-subtitle-stream packet counts from the EXISTING `_count_sub_packets()` (~L3071) on the source + the source duration. Count once per job and reuse the result for MP-10's source-side density if that is a small, local change (otherwise leave MP-10's own count as is and note it). If counting fails (None), the picker falls back to labels only (never crashes).
- D2 Label classes: SIGNS labels = every current keyword EXCEPT the dialogue labels; DIALOGUE labels = "hearing impaired", "closed captions", "sdh", "hi", "cc", "text", plus titles that are exactly/mainly "English", "Full", "Dialogue", "English [CR]"-style. DIALOGUE single words match on word boundaries (\b, case-insensitive); SIGNS matching unchanged (substring, same multi-word-first priority). A track can never be picked as signs because of a DIALOGUE label.
- D3 Tier order for DUBBED, LIVE_ACTION, MOVIES:
T1 disposition.forced=1 → pick (even if dense — typesetting). Reason "disposition".
T2 SIGNS label → pick, scanning ALL English/untagged tracks first by label priority (fixes the ordering bug). Reason "title_keyword=<kw>".
T3 DENSITY (tracks with no SIGNS label/flag, only when ≥2 English/untagged tracks and counts available): pick the sparsest track if its events/min ≤ PICK_SIGNS_MAX_EPM (4.0) AND ≤ PICK_SIGNS_MAX_RATIO (0.5) × the densest English track of the SAME codec family (text = ass/ssa/subrip/webvtt/mov_text; image = hdmv_pgs_subtitle/dvd_subtitle/dvb_subtitle). Reason "density_sparse=<epm>".
T4 no signs found:
DUBBED → OWNER RULE (every English-dub output keeps a default+forced sub for episode titles/foreign scenes): force a dialogue track — prefer a non-SDH/CC track, else the SDH/CC one; reason "dialogue_fallback" (or "single_track" when it is the only English track).
LIVE_ACTION / MOVIES → return (None, "no_signs_track") — never force full dialogue; preset behaviour applies. A lone English track is forced ONLY if T1/T2 matched or it is ≤ 4.0 epm.
- D4 SUBBED → keep today's function behaviour EXACTLY (call the preserved legacy logic) — regression guard; the calibration sample for subbed was too thin (5 files) to change it. Only the word-boundary fix for DIALOGUE labels may differ, and only if a unit test shows the legacy substring match was a false hit.
- D5 Constants PICK_SIGNS_MAX_EPM = 4.0, PICK_SIGNS_MAX_RATIO = 0.5 at module level with a comment citing CALIBRATION_subtitle_density.md.
- D6 MP-10 guard (same change set): in `_SubProfile`, a track with forced=1 or a SIGNS label is `signs_like` regardless of density (typesetting guard), and is never counted as dialogue-like for `full_subs_forced_only_track`; the relative-ratio comparison uses the densest track of the SAME codec family. Keep 0.6 / 4.0 for dialogue_like.
- D7 Log lines keep existing wording for unchanged paths; new reasons log at INFO. Update docstrings (function + module fix history) to describe the tiers, the owner rule and the calibration source.
FIXTURES (server-01 sandbox-fixtures/)
- mp10/: "MP10 D3 - S01E03.mkv" (dub: s:0 full "English", s:1 "Signs & Songs"), "MP10 D4 - S01E04.mkv" (dub: "English" + "English SDH", no signs), "MP10 L1 - S01E01.mkv" (live: one "English SDH"), "MP10 M1 (1993).mkv" (movie: full + "English Forced" forced=1), "MP10 S1/S2/S3 - S02E0x.mkv" (subbed). Read layouts with ffprobe first; report mismatches.
- mp15/ (cut by the main session 2026-09-25): "MP15 Y1 - S01E01.mkv" (Yu Yu Hakusho 001 first 150 s: jpn+eng flac; s:0 untitled ass 20 events, s:1 untitled ass 1 event at 74 s → expected signs by T3), "MP15 B1 - S01E27.mkv" (Bookworm S01E27 BD fansub first 150 s: jpn+eng opus; s:0 "Signs & Songs [EMBER]" forced=1 5249 events, s:1 "Dialogue [GJM]" 5263 events → expected s:0 by T1, and MP-10 must NOT flag it).
- DERIVED fixture you create: `sandbox-fixtures/mp15/MP15 D6 - S01E06.mkv` = D3 remuxed with s:0 retitled "English SDH" (throwaway container on server-01 with the sandbox transcoder image, --entrypoint ffmpeg, mount at /w NEVER /lib: `ffmpeg -i "MP10 D3 - S01E03.mkv" -map 0 -c copy -metadata:s:s:0 title="English SDH" "MP15 D6 - S01E06.mkv"`); ffprobe to confirm.
STEPS
0 Baseline: BASELINE sha; working copy == sandbox container == live; sandbox-input empty; fixtures present; create D6; reset DB; restart transcoder (ELEVATE).
1 OLD-code harness (no transcodes): in a throwaway container or on primary via importlib, ffprobe each fixture (-show_streams -of json) and call the CURRENT picker; record (idx, reason, title) for D3, D4, D6, L1, M1, S1, S2, S3, Y1, B1. EXPECT D6 → SDH (bug), Y1 → none or wrong (untitled), L1 → forced SDH.
2 Implement D1–D7; add `test_mp15_picker.py` (synthetic streams + counts): D6-style SDH-first+signs → signs; typeset-heavy forced signs (dense) → T1 kept; untitled sparse → T3; untitled dense pair → no T3 → dub dialogue_fallback; image vs text family ratio isolation; live lone SDH dense → None; live lone sparse untitled → forced; movie forced → T1; subbed layouts → identical to legacy; word boundaries ("Chinese", "Hindi", "accessibility", "context" never dialogue; "English (CC)", "English [SDH]" are); counts=None → labels-only path; MP-10 guard: forced/signs-labelled dense track = signs_like, no full_subs_forced_only_track. Run ALL unit test files — all pass. py_compile; scp; restart; confirm hash.
3 NEW-code harness over the same 10 fixtures (with real packet counts). EXPECT: D3 s:1 signs; D6 s:1 signs (FIXED); D4 SDH/full via dialogue_fallback (owner rule); L1 None; M1 s:1 T1; S1–S3 identical to step 1; Y1 s:1 density_sparse; B1 s:0 disposition.
4 Sandbox transcodes (end-to-end): D6, D4, Y1, B1 (dubbed), L1 (live), M1 (movies), S2 (subbed) — one category batch at a time. For each output: ffprobe subtitle streams + dispositions, and the job's [VERIFY] lines. EXPECT: D6 forced+default = "Signs & Songs"; D4 forced+default = the dialogue fallback track and verify ≤ warn; Y1 forced+default = the 1-event untitled track; B1 forced+default = "Signs & Songs [EMBER]" and verify has NO full_subs_forced_only_track and no fail; L1 no forced subtitle and verify no fail; M1 forced = "English Forced"; S2 unchanged (pass subbed_dialogue_default). Un-stage/clear between batches; reset DB + restart if the done-set would skip a file.
5 Verdict: GO iff steps 1–4 match every expectation and all unit tests pass. NO promotion. On NO-GO give the failing expectation with evidence.
6 Clean up: sandbox-input back to 4 empty dirs; clear output/renamed/tmp; reset DB; sandbox transcoder RUNNING idle on the new working copy; keep sandbox-fixtures/mp15/* (regression); no leftover waiters; delete your temp scripts.
SCOPE ALLOWLIST: primary working copy media-transcoder.py + new test_mp15_picker.py next to it (and test_verify_output_streams.py ONLY if a D6 guard change requires updating an existing expectation — say which); server-01 sandbox dir contents (script copy, sandbox-input/output/renamed/tmp, sandbox-fixtures/mp15/) + sandbox DB; your temp scripts; the context.md append. Nothing else.
PERSIST BEFORE YOU FINISH
- Append (one `cat >>` heredoc; append only) "## MP-15 v2 — 2026-09-25 (background agent)" to /opt/appdata/docker/docker-compose/media-downloader-local/.claude/context.md: What was done / Decisions / Current state / Next step.
- Run: python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5
- FINAL message = wrap-up JSON only, always:
{"status":"succeeded|partially_succeeded|failed","project":"media_pipeline","phase":"MP-15v2","actions_taken":[],"actions_failed":[],"files_touched":[],"containers_restarted":[],"unverified_claims":[],"baseline_sha256":"","new_sha256":"","old_code_picks":{},"new_code_picks":{},"transcode_results":{"D6":{},"D4":{},"Y1":{},"B1":{},"L1":{},"M1":{},"S2":{}},"mp10_guard":{"B1_codes":[],"D4_codes":[]},"unit_tests":{"passed":null,"failed":null},"verdict":"GO|NO-GO","diff_summary":"","next_step":"","notes":""}
claude-config/config/prompts/media-pipeline/MP-15a_subtitle_density_scan.md
+49
View File
@@ -0,0 +1,49 @@
# MP-15a — subtitle density calibration scan (read-only) — feeds the MP-15 v2 content-first picker
Tracks: personal_projects 61 (fix inventory row 9, MP-15). Owner decision 2026-09-25 18:03: replace keyword-first
forced-sub picking with a content-first picker (subtitle density = events/min), keywords only as tie-breakers.
Written + spawned 2026-09-25.
---
You are a bounded background RESEARCH agent for the media_pipeline project, PHASE MP-15a (read-only calibration scan). Budget: --max-turns 25.
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with status=partially_succeeded.
HARD RULES
- READ-ONLY. Never modify, move, rename or delete any media file. No container restarts/recreates, no config edits, no Postgres writes (read-only SELECTs on the prod media_pipeline DB are allowed: PG=$(docker ps --format '{{.Names}}' | grep '^postgres-'); docker exec "$PG" psql -U postgres -d media_pipeline -c "<one SELECT>"), no git add/commit/push, no sudo, no notifications.
- Run ffprobe only inside a throwaway container with limits, e.g. `docker run --rm --memory 2g --cpus 2 -v "<dir>":/m:ro --entrypoint <ffprobe|python3> <live transcoder image from docker inspect -f '{{.Config.Image}}' media-transcoder-media_transcoder-1> ...` — mount media at /m (NEVER /lib), always `:ro`. Gate: `free -m` MemAvailable >= 6000 on primary before each batch; if lower, wait/stop and report.
- The media share is an external NTFS drive that re-enumerates on I/O stress: keep it gentle — ONE file at a time, no parallel reads, and a hard cap (below). Stop immediately and report if you see I/O errors or "Transport endpoint is not connected".
- Mask secrets. If a hook blocks something, stop with a partial wrap-up.
GOAL
Measure the density (subtitle events per minute of video) of every ENGLISH or untagged subtitle stream in a representative sample of real ORIGINAL source files, so MP-15 v2 can set evidence-based thresholds for "signs/forced-like" vs "dialogue-like" and see how many files fall in a gray zone.
HOW TO MEASURE (be consistent with MP-10)
- Read `/opt/appdata/docker/docker-compose/media-transcoder/media-transcoder.py` (live, sha 7617fc0a): `_count_sub_packets()` (~L3071), `_SubProfile.density/signs_like/dialogue_like` (~L2978-2992) and the constants VERIFY_SIGNS_MAX_RATIO=0.35, VERIFY_DIALOGUE_MIN_RATIO=0.6, VERIFY_DIALOGUE_MIN_PER_MIN=4.0 (~L2801). Use the SAME counting method (packets per subtitle stream / duration minutes) so the thresholds transfer. Also record per stream: codec (ass/subrip/hdmv_pgs_subtitle/dvd_subtitle...), title, language tag, disposition.forced, disposition.default, and — for text codecs only — the total character count if cheaply available (optional; skip if it needs a second full read).
- Note: image-based subs (PGS/VobSub) emit packets for clears too — flag them separately; their densities may not be comparable.
SAMPLE (cap: 150 files total, ONE at a time; prefer breadth over depth — at most 3 files per show/movie)
Source locations (all read-only):
- Seedbox originals: `/media/mediashare/MY MEDIA/MY MEDIA/seedbox local/webdav/Finished/Qbittorrent/` → `Completed Anime/English Dubbed Anime/`, `Completed Anime/English Subbed Anime/`, `Completed Tv Shows/`, `Completed Movies/`.
- Transcode queue originals (kept 14 days): `/media/mediashare/MY MEDIA/MY MEDIA/H264 MEDIA TO BE TRANSCODED/` (4 category dirs).
Aim for ~50 dubbed anime, ~40 subbed anime, ~30 live action, ~30 movies (fewer if not available). Skip `* OP*`, `* ED*`, NCOP/NCED extras. Record the category from the path.
Cross-check: SELECT the prod `pipeline_learning` rows for subtitle_selection (read the table's columns first with information_schema) to see which reasons (disposition / title_keyword=<kw> / single_track) prod has used historically and how often.
ANALYSIS
For each file with ≥1 English/untagged track, classify with ground truth where possible: a track is "labelled signs" if its title contains signs/songs/forced/foreign/"on-screen" style words OR disposition.forced=1; "labelled dialogue" if titled full/English/Dialogue/SDH/CC or untitled in a multi-track file with a labelled signs sibling. Then report:
1. Distribution (min / p10 / median / p90 / max events/min) for labelled-signs vs labelled-dialogue, per category and per codec family (text vs image).
2. Candidate thresholds for the picker: an absolute signs ceiling (events/min) and a relative ratio vs the densest English track in the same file; show how many files each candidate pair classifies correctly vs wrongly, and list the GRAY-ZONE files (neither clearly signs nor clearly dialogue) by name with their numbers.
3. Single-English-track files: how many are sparse (signs-like) vs dense (dialogue-like) per category — this decides tier-3 behaviour.
4. Mislabel evidence: tracks flagged forced or titled signs that are actually dense, and dialogue-titled tracks that are actually sparse.
5. A RECOMMENDED threshold set (absolute + relative) with a one-line rationale, and whether MP-10's existing constants (0.35 / 0.6 / 4.0) should be reused as-is or adjusted.
OUTPUT
- /opt/appdata/docker/docker-compose/media-transcoder/design/CALIBRATION_subtitle_density.md (summary + tables + gray-zone list + recommendation) and a CSV next to it: CALIBRATION_subtitle_density.csv (one row per subtitle stream: file, category, stream idx, codec, lang, title, forced, default, packets, duration_min, events_per_min, ratio_to_densest).
SCOPE ALLOWLIST: create those two files; your temp scripts in your own scratch dir (delete at end); the context.md append. Nothing else.
PERSIST BEFORE YOU FINISH
- Append (one `cat >>` heredoc; append only) "## MP-15a density scan — 2026-09-25 (background agent)" to /opt/appdata/docker/docker-compose/media-downloader-local/.claude/context.md: What was done / Recommended thresholds / Next step.
- Run: python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5
- FINAL message = wrap-up JSON only, always:
{"status":"succeeded|partially_succeeded|failed","project":"media_pipeline","phase":"MP-15a","actions_taken":[],"actions_failed":[],"files_touched":[],"containers_restarted":[],"unverified_claims":[],"files_scanned":null,"streams_measured":null,"per_category_counts":{},"signs_epm":{"median":null,"p90":null,"max":null},"dialogue_epm":{"min":null,"p10":null,"median":null},"recommended":{"abs_signs_max_epm":null,"rel_signs_max_ratio":null,"dialogue_min_epm":null,"reuse_mp10_constants":null},"gray_zone_files":[],"single_track_split":{},"mislabels":[],"prod_reason_history":{},"report":"","next_step":"","notes":""}
claude-config/config/prompts/media-pipeline/MP-18_deinterlace_build.md
+63
View File
@@ -0,0 +1,63 @@
# MP-18 deinterlace build — exact prompt as spawned (2026-09-25)
Tracks: personal_projects 61. Recovered verbatim from the session transcript 2026-09-25 (it was spawned inline and not saved at the time).
---
You are a bounded background agent for the media_pipeline project. Budget: --max-turns 40 (reason: edit + unit tests + 2 old/new sandbox runs + a primary VAAPI check).
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with status=partially_succeeded.
HARD RULES
- NEVER restart, stop, recreate, or exec a mutating command in ANY live/prod container, on any host. Read-only `docker exec ... sha256sum|ffprobe|cat` and `docker ps/inspect` on live containers are allowed.
- Container restart allowlist starts EMPTY. Before you start/stop/restart a SANDBOX container, add it to your allowlist and log "ELEVATED <name>" in actions_taken; verify it is healthy after, and list it in containers_restarted. Only `media-pipeline-sandbox-*` containers on server-01 may ever be elevated. Throwaway `docker run --rm` containers you create yourself are fine.
- NO git add/commit/push. NO writes to the primary host's Postgres. The main session does all DB bookkeeping.
- Issue every mutating command as its OWN tool call; verify with a SEPARATE call (never mutate+sleep+verify in one `sh -c`).
- WAITING: never hand-write nested-quote wait loops. Write the wait condition as a small script FILE on the target host (SQL in a .sql file run with `psql -f`), run it with `timeout 900`. Before your wrap-up, confirm no leftover waiters on both hosts: `ps -eo pid,args | grep "[t]imeout"` (kill only ones you started).
- Mask secrets in all output. You need NO sudo. If a security hook/permission BLOCKS an action, do not work around it: emit a partial wrap-up.
- Do not make design decisions. Every decision is pre-made below; if reality contradicts it (code anchor missing, fixture absent, hash mismatch), STOP and report.
- NO PROMOTION to live in this phase. You deliver a verdict; the main session promotes.
ACCESS MAP
- server-01: ssh administrator@192.168.1.90 (batch several read commands per ssh call). 24 cores, ~27 GB free.
- Sandbox dir (server-01, and a git-tracked twin on the primary host at the same path): /opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/
- Sandbox containers: media-pipeline-sandbox-media-transcoder-sandbox-1, media-pipeline-sandbox-postgres-1 (DB media_pipeline_sandbox, user sandbox, no password needed via docker exec), plus media-api-sandbox / media-downloader-local-sandbox (leave those alone unless a reset requires the downloader restart).
- Sandbox baseline reset: `TRUNCATE pipeline_learning, transcode_jobs, transcode_overrides, pipeline_events, download_jobs, rename_jobs, show_configs RESTART IDENTITY;` then pipe init/02-seed.sql: `docker exec -i media-pipeline-sandbox-postgres-1 psql -U sandbox -d media_pipeline_sandbox < init/02-seed.sql`. ALWAYS restart the sandbox transcoder AFTER a reset (it caches the done-set at startup).
- sandbox-output/, sandbox-renamed/, sandbox-tmp/ hold ROOT-owned files: clear them from inside the container (`docker exec media-pipeline-sandbox-media-transcoder-sandbox-1 find /app/<dir> -mindepth 1 -delete` — check the in-container mount paths with docker inspect first), not host rm.
- STAGE a fixture = cp it into sandbox-input/H264 ENGLISH DUBBED ANIME TO BE TRANSCODED/That Time I Got Reincarnated as a Slime/ (path drives the preset — never flatten). sandbox-input/ must contain only the 4 empty category dirs when you finish.
- The sandbox encodes with software libx265 (server-01 GPU is NVIDIA, no VAAPI). Expected.
- Script edit workflow: the primary-host working copy is /opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/media-transcoder.py (currently identical to live). Record the live file's sha256 (/opt/appdata/docker/docker-compose/media-transcoder/media-transcoder.py) as BASELINE — expected prefix c5f69f61. Edit the working copy; `python3 -m py_compile`; scp it to the same path on server-01; restart the sandbox transcoder; confirm `docker exec media-pipeline-sandbox-media-transcoder-sandbox-1 sha256sum /app/media-transcoder.py` equals the working copy's sha256.
- Logs: `ssh administrator@192.168.1.90 "docker logs --since <ts> media-pipeline-sandbox-media-transcoder-sandbox-1 2>&1 | grep -E '<pattern>'"` (INFO level; the full ffmpeg command is logged as "ffmpeg command:").
- MEASURING TOOL: tools/av_align.py (in the sandbox dir on both hosts). Run it inside the transcoder image, e.g. on server-01:
docker run --rm --memory 4g --cpus 4 -v "<src dir>":/s:ro -v "<out dir>":/o:ro -v /opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/tools:/t:ro --entrypoint python3 gitea.local/backtalk6858/media-transcoder:latest /t/av_align.py "/s/<src file>" "/o/<out file>" --points 30,90
(use the sandbox transcoder's own image name from `docker inspect -f '{{.Config.Image}}' media-pipeline-sandbox-media-transcoder-sandbox-1` if that tag differs). It prints JSON: head_frame_offset (exact, 0 = no duplicated/dropped head frames), per-point audio_lag_ms/video_lag_ms. Read its docstring (per-point video lag carries a ~1-frame seek bias; use head_frame_offset for absolute alignment). Mount media at /s,/o — NEVER at /lib.
BACKGROUND (already diagnosed by the main session — do not re-diagnose)
Prod job 3416 (Slime S04E22, progressive 23.976 CR WEB-DL H264 source) came out with 2 duplicate frames at the head → picture 83 ms late vs audio from 0:00, plus soft field-interpolated frames. Cause: preset PictureDeinterlaceFilter "decomb" maps to an UNCONDITIONAL `yadif=mode=1` (field-rate bob, doubles to 47.95 fps) and `-r src_avg_fps` then decimates back to 23.976, inserting the head duplicates. The `-r` comment in build_segment_cmd ("r_frame_rate can be 2x avg_frame_rate … double-speed output") actually describes yadif's doubling.
PRE-MADE DECISIONS (binding)
- D1: In build_segment_cmd, `deinterlace == "decomb"` must append `yadif=mode=send_frame:parity=auto:deint=interlaced` (NOT yadif=mode=1) on BOTH paths: the VAAPI filter list (~line 1873, `vaapi_filters.append("yadif=mode=1")`) and the CPU list (~line 1941, `cpu_filters.append("yadif=mode=1")`). This deinterlaces only frames flagged interlaced and emits one frame per input frame. Update the docstrings that say decomb -> yadif=mode=1 (~L84, ~L1593, ~L1614-1617) to match. Localized edits only.
- D2: Keep the `-r src_avg_fps` branch unchanged.
- D2b (ONLY if the NEW progressive run in step 3 still shows head_frame_offset != 0): replace, on both paths, `cmd += ["-r", src_avg_fps]` with `cmd += [_fps_flag, "passthrough"]` (keep the else-branch as is), update the comment to say why, re-run step 3 and 4. Report that D2b was needed.
- D3: Unit tests live next to the working copy on the primary host: test_vaapi_cmd.py (14 tests, VAAPI build) — update any assertion expecting yadif=mode=1 and ADD tests asserting (a) CPU and VAAPI commands built with PictureDeinterlaceFilter "decomb" contain `yadif=mode=send_frame:parity=auto:deint=interlaced`, (b) no command contains `yadif=mode=1`, (c) "off" adds no yadif. Run test_vaapi_cmd.py, test_verify_output_streams.py and test_av_stage2_suppression.py (python3 -m pytest or unittest, whichever they use) — all must pass.
FIXTURES (cut by the main session; on server-01 in sandbox-fixtures/mp18/)
- `That Time I Got Reincarnated as a Slime - S04E22.mp18-prog.mkv` — 121 s stream copy of the prod source (progressive, 24000/1001, stereo AAC, ASS signs + SRT CC).
- `That Time I Got Reincarnated as a Slime - S04E22.mp18-interlaced.mkv` — 60 s synthetic interlaced (libx264 +ilme+ildct, field_order tb, 30000/1001), AAC + ASS.
STEPS
Step 0 — Baseline: record BASELINE (live sha256; expect c5f69f61…); confirm the working copy and the sandbox container's /app/media-transcoder.py have the same hash; confirm sandbox-input/ holds only the 4 empty category dirs; confirm both fixtures exist. Reset the sandbox DB baseline and restart the transcoder (ELEVATE first). Clear sandbox-output/renamed/tmp.
Step 1 — Reproduce on OLD code: stage mp18-prog only; wait for its transcode_jobs row status=done (SQL file + timeout script). Find the output file (sandbox-output/ or sandbox-renamed/ — check the job row's output_path, mapped to the host path). Run av_align.py (source = the fixture, --points 30,90). EXPECT head_frame_offset = 2 (bug reproduced). Also record output avg_frame_rate and packet count (`ffprobe -v error -select_streams v:0 -count_packets -show_entries stream=nb_read_packets,avg_frame_rate`) vs the source's. Un-stage; clear outputs; reset DB; restart transcoder.
Step 2 — Implement D1 (+docstrings) and D3 in the working copy; py_compile; run the 3 unit test files; scp; restart the sandbox transcoder; confirm in-container hash. Capture the unified diff of your hunks only.
Step 3 — NEW run A (progressive): stage mp18-prog; wait done. EXPECT: logged ffmpeg command contains `yadif=mode=send_frame:parity=auto:deint=interlaced` and not `yadif=mode=1`; head_frame_offset = 0 with head_corr >= 0.95; output video packet count within ±1 of the source's; avg_frame_rate 24000/1001; audio_lag_ms at t=30 and t=90 within ±15 ms. If head_frame_offset != 0 → apply D2b and redo steps 2–3 once. Un-stage; clear; reset; restart.
Step 4 — NEW run B (interlaced): stage mp18-interlaced; wait done. EXPECT output avg_frame_rate 30000/1001 (NOT 60000/1001); packet count within ±1 of source; the output is deinterlaced: run `ffmpeg -v info -i <file> -vf idet -frames:v 600 -f null - 2>&1 | grep 'Multi frame detection'` on the SOURCE (expect TFF/BFF dominant — validates the fixture) and on the OUTPUT (expect Progressive >= 90% of the counted frames). head_frame_offset = 0 (av_align works on 29.97 too). Un-stage; clear.
Step 5 — VAAPI path check on the PRIMARY host (the only host with VAAPI: AMD /dev/dri/renderD128). Gate: `free -m` MemAvailable >= 6000 MB, else skip this step and mark vaapi_check "skipped_low_ram". scp both fixtures from server-01 into /tmp/mp18/ on primary. In a throwaway container (`docker run --rm --memory 4g --cpus 4 --device /dev/dri/renderD128 -v /tmp/mp18:/w -v <primary working-copy dir>:/c:ro --entrypoint python3 <live transcoder image from docker inspect -f '{{.Config.Image}}' media-transcoder-media_transcoder-1> ...`) import the working copy (/c/media-transcoder.py via importlib), set its module-level USE_VAAPI = True (and anything detect_vaapi would set — read the code), build the segment command for each fixture exactly the way transcode_file_chunked does (read it: probe → video_info/streams → preset for the DUBBED ANIME path — pass an input path under a dir named "H264 ENGLISH DUBBED ANIME TO BE TRANSCODED" if preset selection is path-derived, e.g. by symlinking inside /w), run it with the parts pattern pointing into /w/out-<name>/, then run av_align.py on part_0000.mkv vs the fixture. EXPECT for prog: head_frame_offset 0, 24000/1001, packet count ±1; for interlaced: 30000/1001 and idet Progressive >= 90%. Do NOT touch the live transcoder container. Delete /tmp/mp18 when done.
Step 6 — Verdict GO iff: step 1 reproduced head_frame_offset=2, steps 3 and 4 met every expectation, step 5 met expectations (or was skipped for RAM — then verdict "GO-CPU-ONLY"), all unit tests pass. NO promotion. On NO-GO report which expectation failed with evidence.
Step 7 — Clean up: sandbox-input/ back to 4 empty category dirs; clear sandbox-output/renamed/tmp; leave the sandbox transcoder RUNNING idle with the NEW working copy loaded (it stays the sandbox's copy); no leftover waiters; delete any temp script files you created (by name).
SCOPE ALLOWLIST: primary-host working copy media-transcoder.py and the 3 unit test files next to it (test_vaapi_cmd.py may be edited; add tests there or in a new test_mp18_deinterlace.py in the same dir); server-01 sandbox dir (script copy, sandbox-input/, sandbox-output/, sandbox-renamed/, sandbox-tmp/, sandbox DB); /tmp/mp18 on primary; your own wait-script files; the context.md append below. Nothing else. Do NOT edit the live media-transcoder.py.
PERSIST BEFORE YOU FINISH
- Append (one `cat >>` heredoc) a dated block "## MP-18 — 2026-09-25 (background agent)" to /opt/appdata/docker/docker-compose/media-downloader-local/.claude/context.md: What was done / Decisions / Current state / Next step. Another agent may append to the same file concurrently — append only, never rewrite it.
- Run: python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5
- Your FINAL message is the wrap-up JSON only, always (success or failure):
{"status":"succeeded|partially_succeeded|failed","project":"media_pipeline","phase":"MP-18","actions_taken":[],"actions_failed":[],"files_touched":[],"containers_restarted":[],"unverified_claims":[],"baseline_sha256":"","new_sha256":"","bug_reproduced_on_old_code":{"head_frame_offset":null,"out_fps":"","pkt_src":null,"pkt_out":null},"run_a_prog":{"head_frame_offset":null,"head_corr":null,"out_fps":"","pkt_src":null,"pkt_out":null,"audio_lag_ms":[]},"run_b_interlaced":{"out_fps":"","pkt_src":null,"pkt_out":null,"idet_src":"","idet_out":"","head_frame_offset":null},"vaapi_check":{"status":"pass|fail|skipped_low_ram","prog":{},"interlaced":{}},"d2b_needed":false,"unit_tests":{"passed":null,"failed":null},"verdict":"GO|GO-CPU-ONLY|NO-GO","diff_summary":"","next_step":"","notes":""}
claude-config/config/prompts/media-pipeline/MP-19_chunk_join_drift_design.md
+46
View File
@@ -0,0 +1,46 @@
# MP-19 chunk join drift design — exact prompt as spawned (2026-09-25)
Tracks: personal_projects 61. Recovered verbatim from the session transcript 2026-09-25 (it was spawned inline and not saved at the time).
---
You are a bounded background agent for the media_pipeline project. PHASE MP-19 — DESIGN ONLY (write a design doc, then STOP). Budget: --max-turns 30.
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with status=partially_succeeded.
HARD RULES
- NEVER restart, stop, recreate, or exec a mutating command in ANY existing container on any host (live OR sandbox). Another agent (MP-18) is using the server-01 media-pipeline sandbox right now: do NOT touch any `media-pipeline-sandbox-*` container, its DB, or its sandbox-input/output/renamed/tmp dirs. Read-only `docker ps/inspect` is fine.
- You MAY create throwaway `docker run --rm --memory 6g --cpus 8 ...` containers on server-01 only, working in a scratch dir `~/mp19-scratch` (create it; delete it at the end). Do not run encodes on the primary host (RAM-constrained).
- NO code edits to any media-transcoder.py (live or working copy). NO git add/commit/push. NO Postgres writes anywhere. No notifications, no deploys, no sudo.
- Issue every mutating command as its own tool call. Never hand-write nested-quote wait loops; if you must wait, run the long job in the foreground with `timeout`. Before wrap-up confirm no leftover processes you started (`ps -eo pid,args | grep "[t]imeout"`, `docker ps` for your throwaways).
- Mask secrets. If a security hook blocks an action, do not work around it: partial wrap-up.
- Design choices are yours to RECOMMEND (that is the deliverable), but measure, don't guess.
ACCESS
- server-01: ssh administrator@192.168.1.90 (24 cores, ~27 GB free). Batch commands per ssh call.
- Image with ffmpeg 6.1.2 + python3 + numpy: gitea.local/backtalk6858/media-transcoder:latest (confirm with `docker images | grep media-transcoder` on server-01; use whatever tag the sandbox transcoder uses: `docker inspect -f '{{.Config.Image}}' media-pipeline-sandbox-media-transcoder-sandbox-1`). Run ffmpeg with --entrypoint ffmpeg or sh. Mount media at /w etc — NEVER at /lib.
- Fixture (read-only; copy it into your scratch dir first): server-01 /opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/sandbox-fixtures/mp19/That Time I Got Reincarnated as a Slime - S04E22.mp19-joins.mkv — 330 s stream copy of a prod source (progressive H264 24000/1001, stereo AAC, ASS signs + SRT CC subs, font attachments).
- Measuring tool: /opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/tools/av_align.py (on server-01). Read its docstring. Usage: docker run --rm -v <dir>:/w -v <tools dir>:/t:ro --entrypoint python3 <image> /t/av_align.py /w/<source> /w/<output> --points 30,55,65,... → JSON with head_frame_offset and per-point audio_lag_ms / video_lag_ms (+ = output late vs source; per-point video lag carries a constant ~1-frame seek bias — compare points to each other).
- Code to read (primary host, read-only): /opt/appdata/docker/docker-compose/media-transcoder/media-transcoder.py — build_segment_cmd (both the VAAPI and CPU paths end with `-f segment -segment_time <seg_len> -reset_timestamps 1 part_%04d.mkv`), transcode_file_chunked (~L3028; seg_len = SEGMENT_SECONDS, prod env = 300), write_concat_list (~L2380), the assembly block (`# ---- Assemble final file via concat demuxer ----`, concat demuxer + `-c copy`, or audio re-encode when an A/V correction is applied), verify_playback's "non monotonically increasing dts" filter, and anything using state.json / parts for resume (docstring line ~92 claims "resume / debugging on failure" — determine whether resume is actually implemented or parts are used for anything else, e.g. part validation, progress, crash recovery).
BACKGROUND (diagnosed by the main session from prod job 3416, Slime S04E22, 1440 s, SEGMENT_SECONDS=300)
Audio vs source: +7 ms until the first join, then +167, +225, +293, +355 ms after joins at 300/600/900/1200 s; audio has holes of 160/58/67/63 ms at those joins (packet scan). Video steps 3/1/2/1 frames at the same joins. Net A/V error wobbles ≈ ±40 ms per join and cumulative content delay reaches ~0.4 s. Mechanism hypothesis: each part is muxed with reset timestamps and the concat demuxer starts part N+1 at part N's container duration (max of its streams), so each stream gets shifted by (part duration − its own length); libopus priming/pre-skip and AAC frame boundaries make audio and video part lengths differ. Separately, MP-18 (another agent) is fixing a yadif=mode=1 + `-r` head-duplication bug — to isolate join effects, your prototypes must NOT use yadif and should keep `-r 24000/1001` exactly as prod does (the fixture is progressive).
TASK
1. Read the code; write down the exact current segment + assembly commands (from build_segment_cmd for the DUBBED ANIME CPU path: libx265 -crf 24 -preset veryfast -x265-params log-level=error:pools=6:tune=grain -r 24000/1001 -vf scale=w='min(1920,iw)':h='min(1080,ih)':force_original_aspect_ratio=decrease -c:a libopus -b:a 160k -ac 8 -c:s copy -disposition:s:0 default+forced -map 0:v:0 -map 0:a:0 -map 0:s:0 — confirm from the code) and answer: is resume implemented? what else depends on parts?
2. REPRODUCE: run the current pipeline on the fixture with -segment_time 60 (→ 5 joins at ~60/120/180/240/300 s), concat -c copy exactly like the code. Measure with av_align (--points 30,55,65,115,125,175,185,235,245,295,305,320), plus an audio packet gap scan (gaps > 30 ms: ffprobe -select_streams a:0 -show_entries packet=pts_time,duration_time) and a video packet-count vs source. Confirm the drift pattern (cumulative audio lag steps at joins). If it does NOT reproduce, say so and stop at a partial design.
3. PROTOTYPE and measure each the same way (same fixture, same encoder args, -segment_time 60 where segmentation applies):
A) video-only segments (-an in the segment encode; subs handled as today or muxed at assembly), plus ONE full-length audio encode of the source (same libopus args) and assemble with concat(video parts) + the audio file + subs from source, -c copy.
B) keep A+V segments but make the concat placement video-based: concat list `duration <video duration of part>` (and/or `inpoint`/`outpoint`) directives per part.
C) no segmentation at all (single output file, same args).
Optionally D) anything you find clearly better (e.g. `-segment_time_delta`, `-copyts` without -reset_timestamps + concat with outpoints) — only if measured.
For each: head_frame_offset, max |audio_lag delta across the clip|, per-join A/V change, audio gap count/size, video packet count vs source, output duration vs source, wall-clock encode time, and what the option breaks or preserves (parts validation, resume, per-part retry, forced-sub disposition re-apply, the A/V-correction audio re-encode path at assembly, MP-10 verify_output_streams expectations, VAAPI path — segmentation is muxer-level so note whether VAAPI differs).
4. Write /opt/appdata/docker/docker-compose/media-transcoder/design/DESIGN_chunk_join_drift.md: Problem (with the prod job 3416 numbers above) / Current code (commands, line anchors) / Reproduction results table / Options A–C(–D) table with measurements / RECOMMENDATION with rationale / exact code change plan for a BUILD phase (functions, lines, new constants, interplay with the A/V correction re-encode at assembly and with MP-18's filter change) / Sandbox test plan for the BUILD (fixture = mp19-joins, SEGMENT_SECONDS override value, pass criteria as numbers) / Risks / Open questions for the owner (only real ones). Mark anything not measured as UNVERIFIED.
5. Delete ~/mp19-scratch on server-01 and any throwaway containers.
SCOPE ALLOWLIST: create DESIGN_chunk_join_drift.md; server-01 ~/mp19-scratch (create + delete); throwaway containers you start; the context.md append below. Nothing else.
PERSIST BEFORE YOU FINISH
- Append (one `cat >>` heredoc — another agent may append concurrently, append only) a dated block "## MP-19 — 2026-09-25 (background agent, design)" to /opt/appdata/docker/docker-compose/media-downloader-local/.claude/context.md: What was done / Recommendation / Current state / Next step.
- Run: python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5
- Your FINAL message is the wrap-up JSON only, always:
{"status":"succeeded|partially_succeeded|failed","project":"media_pipeline","phase":"MP-19","actions_taken":[],"actions_failed":[],"files_touched":[],"containers_restarted":[],"unverified_claims":[],"resume_implemented":null,"parts_dependencies":[],"reproduced":{"yes":null,"audio_lag_steps_ms":[],"gaps_ms":[]},"options":{"A":{},"B":{},"C":{}},"recommendation":"","design_doc":"","owner_questions":[],"next_step":"","notes":""}
claude-config/config/prompts/media-pipeline/MP-19b_R2_build.md
+64
View File
@@ -0,0 +1,64 @@
# MP-19b R2 build — exact prompt as spawned (2026-09-25)
Tracks: personal_projects 61. Recovered verbatim from the session transcript 2026-09-25 (it was spawned inline and not saved at the time).
---
You are a bounded background agent for the media_pipeline project. PHASE MP-19b BUILD — implement design R2 in the SANDBOX, verify, deliver a verdict. Budget: --max-turns 55 (reason: multi-function code change + unit tests + 5 sandbox runs incl. a real mid-job restart + a primary VAAPI check).
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with status=partially_succeeded.
HARD RULES
- NEVER restart, stop, recreate, or exec a mutating command in ANY live/prod container, on any host. Read-only `docker exec ... sha256sum|ffprobe|cat` and `docker ps/inspect` on live containers are allowed.
- Container restart allowlist starts EMPTY. Before you start/stop/restart a SANDBOX container, add it to your allowlist and log "ELEVATED <name>" in actions_taken; verify healthy after; list it in containers_restarted. Only `media-pipeline-sandbox-*` containers on server-01 may be elevated. Throwaway `docker run --rm` containers you create are fine.
- NO git add/commit/push. NO writes to the primary host's Postgres. NO edits to the LIVE media-transcoder.py or the LIVE compose file. NO promotion — verdict only; the main session promotes.
- Issue every mutating command as its OWN tool call; verify with a SEPARATE call.
- WAITING: never hand-write nested-quote wait loops. Write wait conditions as small script FILES on the target host (SQL in a .sql file run with `psql -f`), run with `timeout 900`. Before wrap-up confirm no leftover waiters on both hosts (`ps -eo pid,args | grep "[t]imeout"`; kill only your own).
- Mask secrets. No sudo. If a security hook BLOCKS an action, do not work around it: partial wrap-up.
- Do not make design decisions beyond what the design doc + the owner decisions below say. If reality contradicts them, STOP and report.
OWNER DECISIONS (2026-09-25, binding)
- R2 APPROVED (design: /opt/appdata/docker/docker-compose/media-transcoder/design/DESIGN_resumable_no_drift.md — READ ALL OF IT FIRST, especially §6 recommendation, §7 code change plan, §8 test plan, §9 risks). Option C stays on hold.
- Chunk length = 180 s (3 min). Code default for SEGMENT_SECONDS becomes 180; the SANDBOX compose's SEGMENT_SECONDS value may be changed to 180 (that line only). Do NOT touch the prod compose — the main session changes prod at promotion.
- A job interrupted by SIGTERM (container stop/restart) must stay 'queued' (no failed row, no ntfy, not counted toward MAX_RETRY_FAILURES), per design.
- Sandbox-only test switch APPROVED: env FORCE_AUDIO_DELAY_MS (unset/empty = no effect, which is the prod state) forces audio_delay_ms to that value to exercise the >=150 ms A/V-correction path. Log at WARNING when it is active. It may be set in the SANDBOX only, only for the correction test run, and removed after.
- Also from design MP-19 (DESIGN_chunk_join_drift.md) option B: the concat duration-list formula `duration_N = (v_end_N - st_N) - (v_first_{N+1} - st_{N+1})` (R2 uses a B-style list — follow DESIGN_resumable_no_drift.md §7 exactly).
ACCESS MAP
- server-01: ssh administrator@192.168.1.90 (batch read commands per ssh call). 24 cores.
- Sandbox dir (server-01, and a git-tracked twin on the primary host at the same path): /opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/
- Sandbox containers: media-pipeline-sandbox-media-transcoder-sandbox-1, media-pipeline-sandbox-postgres-1 (DB media_pipeline_sandbox, user sandbox, no password via docker exec).
- Baseline reset: `TRUNCATE pipeline_learning, transcode_jobs, transcode_overrides, pipeline_events, download_jobs, rename_jobs, show_configs RESTART IDENTITY;` then `docker exec -i media-pipeline-sandbox-postgres-1 psql -U sandbox -d media_pipeline_sandbox < init/02-seed.sql`; ALWAYS restart the sandbox transcoder after a reset (done-set cached at startup).
- sandbox-output/, sandbox-renamed/, sandbox-tmp/ hold ROOT-owned files: clear from inside the container (`docker exec media-pipeline-sandbox-media-transcoder-sandbox-1 find <in-container path> -mindepth 1 -delete`; get paths via docker inspect). BUT for the resume test you must NOT clear sandbox-tmp between the kill and the resume.
- STAGE = cp a fixture into sandbox-input/H264 ENGLISH DUBBED ANIME TO BE TRANSCODED/That Time I Got Reincarnated as a Slime/ (path drives the preset). sandbox-input/ must hold only the 4 empty category dirs at the end.
- Sandbox = software libx265 (server-01 GPU is NVIDIA, no VAAPI). Expected.
- Script workflow: the primary-host working copy /opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/media-transcoder.py is currently identical to live (sha 8c111273…; record the live sha256 as BASELINE and confirm). Edit the working copy; `python3 -m py_compile`; scp to the same path on server-01; restart the sandbox transcoder; confirm the in-container sha256 matches. If you change the sandbox compose (SEGMENT_SECONDS / FORCE_AUDIO_DELAY_MS), edit the primary twin and scp it, then recreate ONLY the sandbox transcoder with `docker compose -f <sandbox compose> up -d media-transcoder-sandbox` on server-01 (sandbox has no Vault secrets; this is allowed for the sandbox only) and confirm the env with docker inspect (keys only).
- Logs: `ssh administrator@192.168.1.90 "docker logs --since <ts> media-pipeline-sandbox-media-transcoder-sandbox-1 2>&1 | grep -E '<pattern>'"`.
- Unit tests next to the working copy on primary: test_vaapi_cmd.py (17), test_verify_output_streams.py (36), test_av_stage2_suppression.py (5). Update any assertion that the design invalidates (e.g. -f segment expectations); ADD tests for: deterministic job-dir naming (same input → same dir; different size/mtime → different), part validation (.tmp never counted), plan_hash mismatch → fresh start, SIGTERM → stays queued, FORCE_AUDIO_DELAY_MS unset = no effect, duration-list math on synthetic numbers. All must pass.
- Measuring tool: tools/av_align.py in the sandbox dir on both hosts (read its docstring; head_frame_offset is exact; per-point video lag has a ~1-frame seek bias; points outside the clip are skipped). Run inside the transcoder image with media mounted at /s,/o (NEVER /lib), e.g. `docker run --rm --memory 4g --cpus 4 -v <src dir>:/s:ro -v <out dir>:/o:ro -v <sandbox>/tools:/t:ro --entrypoint python3 <sandbox transcoder image> /t/av_align.py /s/<src> /o/<out> --points ...`.
FIXTURES (server-01 sandbox-fixtures/)
- mp19/That Time I Got Reincarnated as a Slime - S04E22.mp19-joins.mkv (330 s progressive, stereo AAC, ASS signs + SRT CC, fonts; 7913 video packets).
- mp18/...S04E22.mp18-prog.mkv (121 s) and mp18/...S04E22.mp18-interlaced.mkv (60 s 29.97i) — regression for MP-18's deinterlace.
PASS CRITERIA (from the designs; all must hold per run unless noted)
1 head_frame_offset == 0. 2 across av_align points, max-min audio_lag_ms <= 10 and every audio_lag_ms <= 15. 3 audio packet discontinuities |gap| > 3 ms: 0 (ffprobe -select_streams a:0 -show_entries packet=pts_time,duration_time). 4 video packets out == source; no video pts gap > 1.5 frame durations. 5 |format duration delta| <= 50 ms. 6 `ffmpeg -v error -i OUT -f null -` gives 0 lines and the job's verify_output_playback passes. 7 s:0 disposition default=1 forced=1; sign-sub pts within ±5 ms of source. 8 MP-10 verify_output_streams: no fail codes. 9 job completes with no part_* / .tmp leftovers and its job dir cleaned on success.
Points for mp19-joins with 180 s parts: --points 30,170,190,290,320.
STEPS
0 Baseline: record BASELINE; confirm working copy == sandbox container == live hash; sandbox-input empty; fixtures present; reset DB; restart transcoder (ELEVATE).
1 Implement design §7 in the working copy (+ docstrings, + the 180 s default, + FORCE_AUDIO_DELAY_MS, + SIGTERM→queued). py_compile; run ALL unit tests (add the new ones). scp; set sandbox SEGMENT_SECONDS=180; recreate/restart sandbox transcoder; confirm hash + env. Capture a concise diff summary (functions changed, line counts).
2 Run U (uninterrupted): stage mp19-joins; wait done; check criteria 1-9. Record wall time. Un-stage; clear outputs; reset DB; restart.
3 Run K (real mid-job restart): stage mp19-joins; wait until the log/tmp shows at least 1 completed video part and the 2nd part in progress (poll the job dir from the host with a script); then `docker restart media-pipeline-sandbox-media-transcoder-sandbox-1` (its own tool call, ELEVATED). EXPECT: the interrupted job is NOT logged failed (check transcode_jobs + logs); after restart the transcoder re-picks the file, logs that it resumed and skipped the completed part(s), and finishes. Check criteria 1-9 on the output, and that av_align points on both sides of the resume seam match run U within 2 ms. Un-stage; clear; reset; restart.
4 Run C (A/V-correction path): set FORCE_AUDIO_DELAY_MS=200 in the sandbox compose, recreate the sandbox transcoder, stage mp18-prog; wait done. EXPECT the correction log line and a FLAT audio_lag of 200±5 ms at --points 20,60,100 (criteria 1,3,4,5,6,7 still hold; criterion 2 replaced by the flat-200 check). Then REMOVE FORCE_AUDIO_DELAY_MS from the sandbox compose, recreate, confirm the env no longer has it. Un-stage; clear; reset; restart.
5 Run R (regression): stage mp18-prog then mp18-interlaced (one after the other). EXPECT MP-18 behaviour intact: prog head_frame_offset 0 and packets exact; interlaced output 30000/1001, packets exact, idet on output Progressive >= 90%.
6 VAAPI check on PRIMARY (only host with AMD VAAPI /dev/dri/renderD128). Gate: `free -m` MemAvailable >= 6000, else mark skipped_low_ram. scp mp19-joins to /tmp/mp19b/ on primary. In a throwaway container (`docker run --rm --memory 4g --cpus 4 --device /dev/dri/renderD128 ... <live transcoder image from docker inspect -f '{{.Config.Image}}' media-transcoder-media_transcoder-1>`) import the working copy, force USE_VAAPI=True, and run the R2 encode path for mp19-joins exactly as transcode_file_chunked would (read how; symlink the fixture under a dir named "H264 ENGLISH DUBBED ANIME TO BE TRANSCODED/<Show>/" for path-derived preset selection), writing into /tmp/mp19b/. Check criteria 1-7 on the result. Do NOT touch the live container. Delete /tmp/mp19b.
7 Verdict: GO iff runs U, K, C, R pass, VAAPI passes (or GO-CPU-ONLY if skipped for RAM), and all unit tests pass. NO promotion. On NO-GO give the failing criterion with evidence.
8 Clean up: sandbox-input back to 4 empty dirs; clear sandbox-output/renamed/tmp; sandbox compose left with SEGMENT_SECONDS=180 and NO FORCE_AUDIO_DELAY_MS; sandbox transcoder RUNNING idle on the new working copy; no leftover waiters; delete your temp script files by name.
SCOPE ALLOWLIST: primary working copy media-transcoder.py + the unit test files next to it (new test file allowed, e.g. test_mp19b_resume.py); sandbox docker-compose.yml (SEGMENT_SECONDS line and the temporary FORCE_AUDIO_DELAY_MS line only) on primary twin + server-01; server-01 sandbox dir contents + sandbox DB; /tmp/mp19b on primary; your temp scripts; the context.md append. Nothing else.
PERSIST BEFORE YOU FINISH
- Append (one `cat >>` heredoc; append only — other agents may write concurrently) a dated block "## MP-19b BUILD — 2026-09-25 (background agent)" to /opt/appdata/docker/docker-compose/media-downloader-local/.claude/context.md: What was done / Decisions / Current state / Next step.
- Run: python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5
- FINAL message = wrap-up JSON only, always:
{"status":"succeeded|partially_succeeded|failed","project":"media_pipeline","phase":"MP-19b-BUILD","actions_taken":[],"actions_failed":[],"files_touched":[],"containers_restarted":[],"unverified_claims":[],"baseline_sha256":"","new_sha256":"","unit_tests":{"passed":null,"failed":null},"run_U":{},"run_K":{"interrupted_job_status":"","resumed_log_line":"","parts_skipped":null,"criteria":{}},"run_C":{},"run_R":{},"vaapi_check":{"status":"pass|fail|skipped_low_ram"},"verdict":"GO|GO-CPU-ONLY|NO-GO","diff_summary":"","prod_changes_needed_at_promotion":[],"next_step":"","notes":""}
claude-config/config/prompts/media-pipeline/MP-19b_resumable_design.md
+45
View File
@@ -0,0 +1,45 @@
# MP-19b resumable design — exact prompt as spawned (2026-09-25)
Tracks: personal_projects 61. Recovered verbatim from the session transcript 2026-09-25 (it was spawned inline and not saved at the time).
---
You are a bounded background agent for the media_pipeline project. PHASE MP-19b — DESIGN ONLY (write a design doc, then STOP). Budget: --max-turns 35.
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with status=partially_succeeded.
HARD RULES
- NEVER restart, stop, recreate, or exec a mutating command in ANY existing container on any host (live or sandbox). Do NOT touch any `media-pipeline-sandbox-*` container, its DB, or its sandbox-input/output/renamed/tmp dirs. Read-only `docker ps/inspect` is fine.
- You MAY create throwaway `docker run --rm --memory 6g --cpus 8 ...` containers on server-01 only (you may `docker kill` only containers YOU started — that is how you simulate a restart), working in a scratch dir `~/mp19b-scratch` (create it; delete it at the end — files will be root-owned, so delete them from a throwaway container first). No encodes on the primary host.
- NO code edits to any media-transcoder.py (live or working copy). NO git add/commit/push. NO Postgres writes. No notifications, deploys, or sudo.
- Issue every mutating command as its own tool call. Never hand-write nested-quote wait loops; run long jobs in the foreground with `timeout`, or write a small script file. Before wrap-up confirm no leftover processes/containers you started.
- Mask secrets. If a security hook blocks an action, do not work around it: partial wrap-up.
ACCESS
- server-01: ssh administrator@192.168.1.90 (24 cores, ~27 GB free). Batch commands per ssh call.
- Image (ffmpeg 6.1.2 + python3 + numpy): use `docker inspect -f '{{.Config.Image}}' media-pipeline-sandbox-media-transcoder-sandbox-1` on server-01. Mount media at /w etc — NEVER at /lib.
- Fixture (read-only; copy into your scratch dir first): server-01 /opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/sandbox-fixtures/mp19/That Time I Got Reincarnated as a Slime - S04E22.mp19-joins.mkv (330 s progressive H264 24000/1001, stereo AAC, ASS signs + SRT CC subs, fonts).
- Measuring tool: /opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/tools/av_align.py on server-01 (read its docstring; head_frame_offset exact; per-point video lag has a constant ~1-frame seek bias; points outside the clip are skipped).
- Code (primary host, read-only): /opt/appdata/docker/docker-compose/media-transcoder/media-transcoder.py (live, sha 8c111273). Git history in /opt/appdata/docker (read-only git log/show allowed).
- READ FIRST: /opt/appdata/docker/docker-compose/media-transcoder/design/DESIGN_chunk_join_drift.md (MP-19, done today). It measured the drift mechanism and options A/A2/B/C. Do not redo its measurements; reuse its numbers and its option-B `duration_N` formula.
BACKGROUND (verified by the main session)
- Owner requirement: KEEP the ability to resume a job mid-file after the transcoder container is restarted (the owner relies on it). MP-19 recommended dropping segmentation (C) because it found no resume in today's code; the owner rejected that until a resumable design is proven.
- History: commit a7f88bd (2025-11-09, "pause the file currently being transcoded and then resume transcoding that same file") implemented REAL resume: per-segment ffmpeg runs with -ss/-t, `.partNNNN.mkv` files, state.json, and on restart skipping parts that already exist. Commit ac68e07 (2025-11-27, "error where certain files would fail to transcode would be fixed") rewrote it into today's single-pass `-f segment -segment_time N -reset_timestamps 1` encode with a NEW unique job dir per attempt (make_temp_job_dir: job_<ts>_<pid>) and a write-only state.json, so resume was silently lost. READ both diffs (`git show a7f88bd -- docker-compose/media-transcoder/media-transcoder.py`, same for ac68e07) and work out WHY ac68e07 replaced the per-segment approach (which files failed and how) so the new design does not reintroduce that failure.
- Also determine: after a container restart, how does today's transcoder re-pick the interrupted file (done-set from DB / TranscodeState / transcode_jobs rows; is an in-progress row left behind; does anything clean the old job dir; cleanup_old_failed_job_dirs age rule)? Resume must hook into that path.
- Drift mechanism (MP-19): each part's video starts 58-83 ms after its start_time (x265 reorder delay survives -reset_timestamps); the concat demuxer places the next part at the container duration, inserting that hole into BOTH streams per join. B's duration list removes it.
TASK — design a pipeline that is BOTH resumable and drift-free, and PROVE it:
1. Candidates (measure at least R1 and R2; add others only if measured):
R1) keep today's single-pass segment muxer; make the job dir deterministic per input (e.g. hash of input path + size + mtime); on restart, validate existing parts, drop the last (possibly truncated) one, and restart ffmpeg with input `-ss <exact start of the first missing part>` and `-segment_start_number N`; assemble with option B's duration list. Work out how to get the exact resume timestamp (the next part's first video pts in source time) so the seam has no gap/overlap.
R2) per-part VIDEO-ONLY encodes (-ss/-t on the input, keyframe-aligned boundaries) + ONE full-length audio encode (seconds of work, redone on resume) + subs from source; assemble with a B-style duration list, then mux audio/subs.
2. For each candidate, run: (a) an uninterrupted encode, and (b) a simulated restart: start the encode, `docker kill` the container mid-way through part 3 (-segment_time 60 → kill at ~150 s of media progress), then run the resume logic (a small python/shell prototype script of the resume steps — prototype only, not the transcoder) and assemble. Measure both with av_align.py (--points 30,55,65,115,125,175,185,235,245,295,305,320), an audio packet gap scan (gaps > 3 ms), video packet count vs source (7913), duration delta, sub pts delta, dispositions, and the seam at the resume point specifically. Pass bar = MP-19 design §7 criteria 1-7 for BOTH (a) and (b).
3. Write /opt/appdata/docker/docker-compose/media-transcoder/design/DESIGN_resumable_no_drift.md: Owner requirement / What a7f88bd did and why ac68e07 dropped it / How restart re-picks a file today / Candidates / Results tables (uninterrupted + resumed) / RECOMMENDATION / exact code change plan (functions, line anchors, new state fields, job-dir naming, cleanup-rule changes, VAAPI->CPU retry interplay, A/V-correction finalize interplay, MP-10 interplay) / sandbox BUILD test plan including a real mid-job sandbox-transcoder restart (the BUILD agent may restart the sandbox container) with numeric pass criteria / Risks / Owner questions (only real ones). Mark anything not measured UNVERIFIED. Also add a dated banner at the top of DESIGN_chunk_join_drift.md §5 (one short paragraph): "2026-09-25 owner: resume must be kept → option C on hold; see DESIGN_resumable_no_drift.md".
4. Delete ~/mp19b-scratch and any throwaway containers.
SCOPE ALLOWLIST: create DESIGN_resumable_no_drift.md; the one banner in DESIGN_chunk_join_drift.md; server-01 ~/mp19b-scratch; throwaway containers you start; the context.md append below.
PERSIST BEFORE YOU FINISH
- Append (one `cat >>` heredoc; another agent may append concurrently — append only) a dated block "## MP-19b — 2026-09-25 (background agent, design)" to /opt/appdata/docker/docker-compose/media-downloader-local/.claude/context.md: What was done / Recommendation / Current state / Next step.
- Run: python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5
- Your FINAL message is the wrap-up JSON only, always:
{"status":"succeeded|partially_succeeded|failed","project":"media_pipeline","phase":"MP-19b","actions_taken":[],"actions_failed":[],"files_touched":[],"containers_restarted":[],"unverified_claims":[],"why_ac68e07_dropped_per_segment":"","restart_repick_path":"","candidates":{"R1":{"uninterrupted":{},"resumed":{},"seam":{}},"R2":{"uninterrupted":{},"resumed":{},"seam":{}}},"recommendation":"","design_doc":"","owner_questions":[],"next_step":"","notes":""}
claude-config/config/prompts/media-pipeline/VAAPI_QP_calibration.md
+52
View File
@@ -0,0 +1,52 @@
# VAAPI QP calibration — exact prompt as spawned (2026-09-24)
Tracks: personal_projects 268. Recovered verbatim from the session transcript 2026-09-25 (it was spawned inline and not saved at the time).
---
You are a bounded background agent for the media_pipeline project (personal_projects #268 VAAPI). Budget: --max-turns 25.
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up JSON with
status=partially_succeeded.
HARD RULES
- MEASUREMENT ONLY. Do NOT edit media-transcoder.py or any code/compose (another agent is editing the transcoder now).
NEVER restart/stop/exec-mutate ANY existing container. Do not touch the server-01 sandbox.
- Throwaway containers ONLY on PRIMARY (this host; server-01 has NVIDIA = no VAAPI): `docker run --rm --memory 4g --cpus 4
--device /dev/dri/renderD128 --group-add $(stat -c %g /dev/dri/renderD128)` from image
gitea.local/backtalk6858/media-transcoder:latest (has ffmpeg with hevc_vaapi + libx265). Check `free -m` before each run;
do not start one if MemAvailable < 6 GB (primary is RAM-constrained). Media read-only at /m (NEVER at /lib); outputs only
in the container's /tmp. Run encodes ONE at a time (the live transcoder may start jobs; it has priority). Confirm no test
containers remain at the end.
- NO git, NO DB writes, no notifications, mask secrets, no sudo. Blocked → partial wrap-up.
PERSIST: append "## VAAPI-QP — <date> (background agent)" (What/Findings/State/Next) to
/opt/appdata/docker/docker-compose/media-downloader-local/.claude/context.md (single append); run
python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5. FINAL message = wrap-up JSON only: status,
project ("media_pipeline"), actions_taken[], actions_failed[], files_touched[], containers_restarted[] (empty),
unverified_claims[], next_step, notes + extra fields.
==========================================================================================
PHASE VAAPI-QP — pick the hevc_vaapi QP for the live transcoder. OWNER DECISION (2026-09-24): enable VAAPI because freeing
primary's CPU is worth it, BUT the owner dislikes bigger files → choose the SMALLEST-file setting that keeps quality
acceptable. Rule (pre-decided): pick the highest QP whose mean SSIM vs source ≥ 0.975 on EVERY test clip AND no clip below
0.970; report size ratio vs the current CPU encode (libx265 at the preset CRF). If even the lowest tested QP misses the bar
on a clip type, report that.
Background: /opt/appdata/docker/docker-compose/media-transcoder/design/INVESTIGATION_vaapi.md (read "A/B results" +
"Follow-on blockers"): VCN HEVC is I/P only (no B-frames), CQP only; 1 Slime clip gave QP24 2.6×/0.9803, QP26 1.9×/0.9785,
QP28 1.5×/0.9767 vs CPU CRF24 10.7 MB/0.9807. Encoder min size 256x128. 1080p output decodes as 1920x1088 unless
`-bsf:v hevc_metadata=crop_bottom=8` (crop_bottom = ceil16(h) − h) — USE that bsf on every VAAPI encode.
The live CPU command: read build_segment_cmd() in /opt/appdata/docker/docker-compose/media-transcoder/media-transcoder.py
(read-only) for the exact libx265 args/preset CRF per category (presets english-*.json in that dir) and the VAAPI branch
(-vf format=nv12|p010,hwupload -c:v hevc_vaapi -qp <crf>) — replicate both faithfully (video only; drop audio/subs: -an -sn).
Clips: 5 × 120 s stream-copy excerpts from t=300 s (-c copy), from
"/media/mediashare/MY MEDIA/MY MEDIA/seedbox local/webdav/Finished/Qbittorrent/": 2 dubbed anime (e.g. Yameii Slime S04E06,
Frieren S02E09 — web H264), 1 BluRay x265 anime (Ascendance Headpatter S01E27, 10-bit), 1 subbed (Farming Life S02E09),
1 live action (The Pitt S02E13). For each clip: CPU encode at its category's CRF (record time + size) and VAAPI at QP
24, 26, 28, 30, 32 (time + size). Metrics per encode: SSIM (and PSNR) vs the source excerpt with
`-lavfi "[0:v]format=yuv420p,setpts=PTS-STARTPTS[a];[1:v]format=yuv420p,setpts=PTS-STARTPTS[b];[a][b]ssim"`, AND confirm the
DECODED frame size equals the source (e.g. `-vf showinfo` → s:1920x1080) — never trust ffprobe header size alone. Also note
CPU% of the ffmpeg process for CPU vs VAAPI (e.g. /usr/bin/time -v in the container: user+sys time vs wall).
Output: a table clip × setting (time, size, ratio vs CPU, SSIM, PSNR, decoded WxH, cpu-seconds), the chosen QP per the rule,
and whether a per-category QP (e.g. live action vs anime) would save meaningfully more (> 10 % size) — recommend ONE mapping.
Write the results into a new section "## QP calibration — <date> (VAAPI-QP agent)" APPENDED to INVESTIGATION_vaapi.md (that
file is allowed; append only).
Extra wrap-up fields: {"chosen_qp":null,"per_category_qp":{},"size_ratio_vs_cpu":{},"min_ssim":null,
"decoded_size_ok":true,"cpu_seconds_cpu_vs_vaapi":{},"table_rows":0}
claude-config/config/prompts/media-pipeline/VAAPI_build.md
+99
View File
@@ -0,0 +1,99 @@
# VAAPI build — exact prompt as spawned (2026-09-24)
Tracks: personal_projects 268. Recovered verbatim from the session transcript 2026-09-25 (it was spawned inline and not saved at the time).
---
You are a bounded background agent for the media_pipeline project (personal_projects #268). Budget: --max-turns 40.
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with
status=partially_succeeded.
HARD RULES
- NEVER restart, stop, recreate, or exec a mutating command in ANY live/prod container. Read-only
`docker exec ... sha256sum|ffprobe|cat` and `docker ps/inspect` on live containers are allowed.
- Container restart allowlist starts EMPTY; ELEVATE (log "ELEVATED <name>") before restarting a SANDBOX container; only
`media-pipeline-sandbox-*` on server-01 may ever be elevated; verify healthy after.
- Throwaway test containers on PRIMARY are allowed for the VAAPI tests (server-01 has NVIDIA = no VAAPI):
`docker run --rm --memory 4g --cpus 4 --device /dev/dri/renderD128 --group-add $(stat -c %g /dev/dri/renderD128)`
image gitea.local/backtalk6858/media-transcoder:latest; `free -m` MemAvailable ≥ 6 GB before each; media read-only at
/m (NEVER /lib); outputs only in the container's /tmp; one encode at a time; none left running at the end.
- NO git add/commit/push. NO writes to the primary Postgres. Mask secrets. No sudo. Blocked → partial wrap-up.
- WAIT LOOPS: never hand-write nested-quoted `until [ $(psql -c "...'x'...") ]` one-liners. Put the SQL in a .sql file on
the target host and poll with a small script run under `timeout`; before your wrap-up confirm no leftover waiters on
either host: `ps -eo pid,args | grep "[t]imeout"`.
- Issue every mutating command as its OWN tool call; verify with a SEPARATE call. Do not make design decisions — they are
pre-made below; if reality contradicts them, STOP and report.
ACCESS MAP
- server-01: ssh administrator@192.168.1.90; sandbox dir (server-01 + primary git twin, same path)
/opt/appdata/docker/docker-compose/server-01/media-pipeline-sandbox/ (README.md). Containers (keep healthy):
media-pipeline-sandbox-{postgres,media-transcoder-sandbox,media-api-sandbox,media-downloader-local-sandbox}-1;
filebot-monitor-sandbox stays OFF; sandbox transcoder runs VERIFY_STREAMS_MODE=shadow — leave it.
- Baseline reset: TRUNCATE pipeline_learning, transcode_jobs, transcode_overrides, pipeline_events, download_jobs,
rename_jobs, show_configs RESTART IDENTITY; pipe init/02-seed.sql via docker exec -i media-pipeline-sandbox-postgres-1
psql -U sandbox -d media_pipeline_sandbox < file; then restart media-transcoder-sandbox AND
media-downloader-local-sandbox. ROOT-owned outputs: clear via docker exec
media-pipeline-sandbox-media-transcoder-sandbox-1 find <dir> -mindepth 1 -delete.
- Fixtures: server-01 <sandbox>/sandbox-fixtures/mp10/ ("MP10 D1 - S01E01.mkv", "MP10 L1 offset - S01E02.mkv",
"MP10 M1 (1993).mkv", "MP10 S2 - S02E02.mkv"); stage by cp into sandbox-input/<CATEGORY DIR>/<Show>/ (D → "H264 ENGLISH
DUBBED ANIME TO BE TRANSCODED"/"MP10 D", S → "H264 ENGLISH SUBBED ANIME TO BE TRANSCODED"/"MP10 S", L → "H264 LIVE ACTION
SERIES TO BE TRANSCODED"/"MP10 L", M → "H264 MOVIES TO BE TRANSCODED"/"MP10 M1 (1993)"); ≤3 at a time; input EMPTY at end.
- Script workflow: cp the LIVE /opt/appdata/docker/docker-compose/media-transcoder/media-transcoder.py to the primary sandbox
dir as the working copy (overwrite); BASELINE sha256 must start e3b6f751023fd9f0 (AV-FIX promoted 20:32) — else STOP.
Edit; py_compile; the existing unit tests test_verify_output_streams.py (36) and test_av_stage2_suppression.py (5) in
that dir must still pass; scp to server-01; restart the sandbox transcoder; confirm the container's /app sha256.
- Promotion (only on GO): if the live sha256 still equals BASELINE, cp working copy → live; verify host hash AND
`docker exec media-transcoder-media_transcoder-1 sha256sum /app/media-transcoder.py`. Do NOT restart live. The live
compose already passes /dev/dri/renderD128 (startup log shows "VAAPI: device /dev/dri/renderD128 found").
Human restart command to output: docker restart media-transcoder-media_transcoder-1
PERSIST: append "## VAAPI BUILD — <date> (background agent)" to
/opt/appdata/docker/docker-compose/media-downloader-local/.claude/context.md; run
python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5. FINAL message = wrap-up JSON only:
status, project ("media_pipeline"), actions_taken[], actions_failed[], files_touched[], containers_restarted[],
unverified_claims[], next_step, notes + extra fields.
==========================================================================================
PHASE VAAPI BUILD — make the live transcoder actually use hevc_vaapi on primary's AMD Vega 8 (owner decision 2026-09-24:
enable it — freeing primary's CPU is worth it; owner dislikes bigger files → use the calibrated QPs below).
Read first: /opt/appdata/docker/docker-compose/media-transcoder/design/INVESTIGATION_vaapi.md (root cause, "Proven fix",
"Follow-on blockers", "A/B results", and the "QP calibration — 2026-09-24" section).
PRE-DECIDED CHANGES (working copy only; short functions; comments explain why):
1. detect_vaapi() (~L998): probe with `nullsrc=s=256x256:d=1` (VCN HEVC minimum is 256x128); on failure log the LAST
300 chars of stderr (the first line is a benign amdgpu warning that hid the real error). Do NOT touch the separate
CPU fps_mode probe that also uses 64x64 (~L1062).
2. QP mapping replaces the 1:1 `-qp <crf>` on the VAAPI branch (~L1735): module constants VAAPI_QP_ANIME = 28,
VAAPI_QP_LIVE_ACTION = 32, VAAPI_QP_MOVIES = 28, each overridable by env (VAAPI_QP_ANIME etc.). Category from the input
path exactly like the rest of the script derives it ("LIVE ACTION" → live action, "MOVIES" → movies, else anime — live
action shares the dubbed preset, so the QP must key on the PATH category, not the preset). CPU path/CRF unchanged.
3. Crop fix on every VAAPI encode: `-bsf:v hevc_metadata=crop_bottom=<ceil16(h)-h>:crop_right=<ceil16(w)-w>` (omit a zero
term; omit the bsf entirely when both are 0), where w,h are the OUTPUT frame dims after any filters the command applies.
4. Per-job CPU fallback: sources whose output dims fall outside 256–4096 × 128–4096 use libx265 even when VAAPI is enabled;
and if a hevc_vaapi segment encode returns rc≠0, the job retries ONCE on libx265 (log it; the existing encoder_path
learning row must record the encoder actually used). HDR→CPU fallback already exists — keep it.
5. Deinterlace: LEAVE the existing `yadif=mode=1` behavior exactly as it is (a separate fix, MP-18, will change it — do
not touch it here).
6. MP-10 checks: grep verify_output_streams() for any size/bitrate-ratio check; if one would flag normal VAAPI outputs
(1.2–1.7× CPU size), report it (do not change thresholds) — add a unit test documenting the behavior.
TESTS
a. Unit (new test_vaapi_cmd.py next to the working copy, stdlib unittest): build_segment_cmd for each category on the VAAPI
branch → correct -qp (28/32/28), correct bsf for 1920x1080 (crop_bottom=8), none for 1280x720, 3840x2160 (crop_bottom=0 →
no bsf); a 200x120 source → libx265; detect_vaapi command contains 256x256. All three test files pass.
b. Sandbox CPU regression (server-01, VAAPI impossible there → CPU path): baseline reset; D1, L1 offset, M1 → done exactly
as before (L1 av ≈ −306 ms, M1 ≈ +213 ms, encoder_path libx265, shadow verify rows).
c. Primary VAAPI E2E: in ONE throwaway container on primary (with /dev/dri), import the working copy and run the real
encode path for 3 clips — choose the lightest harness that exercises detect_vaapi() + build_segment_cmd() +
the segment encode + concat (transcode_file_chunked if it runs without a DB/media-api when DB_HOST/MEDIA_API_URL are
empty; otherwise the minimal equivalent — say which). Clips (120 s stream-copy excerpts from t=300 s) from
"/media/mediashare/MY MEDIA/MY MEDIA/seedbox local/webdav/Finished/Qbittorrent/": Yameii Slime S04E06 (dubbed, web H264),
SubsPlease One Piece 1164 (subbed, H264), The Pitt S02E13 (live action). Place each under a temp path containing its
category dir name so the category logic sees it. For each: detect_vaapi()=True, encoder hevc_vaapi, -qp as mapped,
DECODED frame size via `-vf showinfo` equals the source (1920x1080), SSIM vs source ≥ 0.965, audio/subs present as the
CPU path would produce, size vs a CPU encode of the same clip, CPU-seconds. Also one forced-fallback run (env forcing a
VAAPI failure, e.g. an invalid VAAPI_QP or a monkeypatched rc) → job completes on libx265.
GO = a, b, c all pass → PROMOTE. Else NO-GO.
Clean up: sandbox input empty, outputs cleared, DB baseline reset, all 4 sandbox containers healthy, no throwaway containers.
Extra wrap-up fields: {"baseline_sha256":"","working_sha256":"","unit_tests":"","sandbox_cpu_regression":[],
"primary_vaapi":[{"clip":"","encoder":"","qp":null,"decoded_wxh":"","ssim":null,"size_vs_cpu":null,"cpu_s":null}],
"fallback_test":"PASS|FAIL","mp10_size_checks":"","verdict":"GO|NO-GO","promoted":false,"live_sha256_after":"",
"human_restart_cmd":""}
claude-config/config/prompts/wireguard/W1_build_prep.md
+40
View File
@@ -0,0 +1,40 @@
# W1 BUILD-PREP — wg-easy home-access tunnel (#274) — write files, deploy NOTHING
Tracks: personal_projects #274. Design: `/opt/appdata/docker/research/diy_wireguard_design.md` (LOCKED 2026-09-25,
owner accepted all 7 recommendations). Spawned 2026-09-25 from the media-downloader-local conversation.
---
You are a bounded background BUILD-PREP agent for personal_projects #274 (DIY WireGuard home-access tunnel), PHASE W1-PREP. Budget: --max-turns 30.
If you issue the same tool call twice with identical arguments, STOP and output the wrap-up block with status=partially_succeeded.
HARD RULES
- WRITE FILES ONLY. Do NOT start, create, pull or restart any container; do NOT run `docker compose up`; do NOT install, enable or start any systemd unit; no sudo; no router/DNS/Cloudflare changes; no git add/commit/push; no Postgres writes; no ntfy sends. The owner deploys after review.
- Read-only on the host otherwise (docker ps/inspect image/ports/networks/labels only — NEVER .Config.Env; reading files without printing secret values).
- The ONE permitted Vault write: generate the wg-easy admin password straight into Vault with the store tool, never displayed:
`python3 /opt/appdata/docker/.claude/scripts/vault_store.py --target prod wireguard/admin password --generate 24`
(run `--dry-run` first; if the field already exists, do NOT use --replace — reuse it). It prints only a length.
- Mask secrets. If a hook blocks something, stop with a partial wrap-up. Issue each file write as its own call.
LOCKED DECISIONS (from the design + owner, 2026-09-25) — do not revisit
- wg-easy **v15**, pinned to an exact current v15.x tag (look it up on ghcr.io/wg-easy/wg-easy; record the tag and digest), bridge mode on its OWN docker network `wg` (10.42.42.0/24), NOT on `coolify`, NOT host mode. IPv6 off.
- WireGuard UDP port **45791** published on the host (`45791:45791/udp` — the in-container listen port must match; read the v15 docs for the env/UI setting that controls it). Web UI bound to **127.0.0.1:51821 only**; never behind Traefik or cloudflared. Enable TOTP after first login (owner step).
- Endpoint host for clients: `fuppzd3f9v.reverseproxyserver.net` (DNS record created in W2; for W1 LAN testing the owner uses 192.168.1.88:45791).
- Client address range: the design's choice (read it; default 10.8.0.0/24 in wg-easy). Split tunnel: AllowedIPs = the wg-easy client range + 10.42.42.0/24 (+ LAN host /32s only where the design's ACL table allows). Client DNS: leave unset in W1 (dnsmasq arrives in W3).
- Secrets via SecretSpec + Vault, exactly like cloudflared: study `/opt/appdata/docker/docker-compose/cloudflared/` (docker-compose.yml, secretspec.toml, deploy/resolver.sh, deploy/secretspec-resolver.service, deploy/secretspec-resolver.timer) and `playbook_secretspec_env_resolution.md` in /home/administrator/.claude/projects/-opt-appdata-docker/memory/ (READ IT). The admin password lives at Vault `secret/wireguard/admin` field `password` — map it into wg-easy's first-run INIT env (v15 `INIT_ENABLED`/`INIT_USERNAME`/`INIT_PASSWORD` or whatever v15 documents — verify in the docs) through secretspec; username `admin` is not secret. Note: the cloudflared resolver.sh header comment says 'media-downloader-local' (copy-paste leftover) — do not copy that mistake.
- Compose MUST follow the template `/opt/appdata/docker/non-docker-python-scripts/Docker Template/docker-compose.yml` (read it first: field ordering, indentation, comments, section labels). Include healthcheck, restart policy per template, `cap_add: [NET_ADMIN, SYS_MODULE]` only if v15 docs require them (prefer NET_ADMIN only; the wireguard kernel module is already loaded on the host — verify with `lsmod | grep wireguard`), the sysctls v15 documents (ip_forward, src_valid_mark), and a named volume for `/etc/wireguard`.
- #273 image policy: add a proposed entry for wg-easy (tier LATEST — internet-facing) to a NOTE in your README (do NOT edit image-policy.draft.yml).
DELIVERABLES (all under `/opt/appdata/docker/docker-compose/wireguard/`, new dir)
1. `docker-compose.yml` (template format) — wg-easy only for W1 (ddns + dnsmasq are W2/W3; leave commented placeholders naming them).
2. `secretspec.toml` + `deploy/resolver.sh` + `deploy/secretspec-resolver.service` + `deploy/secretspec-resolver.timer` modelled on cloudflared's, with the project name `wireguard`. Unit names: `wireguard-secretspec-resolver.service/.timer`.
3. `README.md`: what W1 is, the exact OWNER deploy steps (install units with sudo, start resolver, verify container healthy, `ss -lnup | grep 45791`, UI at http://127.0.0.1:51821 via SSH tunnel from the laptop: `ssh -L 51821:127.0.0.1:51821 administrator@192.168.1.88`), first login + enable TOTP, create client `laptop`, the LAN test (laptop endpoint 192.168.1.88:45791: handshake, ping the server tunnel IP), the rollback (`systemctl disable --now` the timer/service, `docker compose down`, remove the network), and the W1 verification checklist from design §5. Include the W0 dependency list (NetBird host client disabled; CF tokens are NOT needed for W1).
4. Validate without deploying: `docker compose -f <file> config -q` (warnings about unset secretspec vars are expected — note them), `bash -n deploy/resolver.sh`, `systemd-analyze verify deploy/*.service deploy/*.timer` (read-only analysis).
SCOPE ALLOWLIST: the new `/opt/appdata/docker/docker-compose/wireguard/` directory; the single Vault write above; the context.md append. Nothing else.
PERSIST BEFORE YOU FINISH
- Append (one `cat >>` heredoc; append only) "## #274 W1-prep — 2026-09-25 (background agent)" to "/opt/appdata/docker/Machines/infrastructure general questions/.claude/context.md": What was done / Current state / Owner deploy steps (short) / Next step.
- Run: python3 /opt/appdata/docker/.claude/scripts/embed_memory_dir.py --only-recent 5
- FINAL message = wrap-up JSON only, always:
{"status":"succeeded|partially_succeeded|failed","project":"diy-wireguard #274","phase":"W1-prep","actions_taken":[],"actions_failed":[],"files_touched":[],"containers_restarted":[],"unverified_claims":[],"wg_easy_tag":"","wg_easy_digest":"","vault_admin_password":"stored|reused|failed","validation":{"compose_config":"","resolver_syntax":"","systemd_verify":""},"owner_deploy_steps":[],"next_step":"","notes":""}