claude-projects/agent-builder/.claude/context.md

project_name: agent-builder

Agent Builder — Session Context

⚠️ READ THIS BLOCK ONLY. Everything below "## Architecture" is APPEND-ONLY HISTORY, oldest first, and much of it is superseded. This file is injected into every session by the session-start hook, so treat the rest as an archive to grep, not context to absorb. Do not infer current state from it — it is 568 lines of June sprint-day facts with the oldest on top. Fix pending: prune it (task #192). See feedback-superseding-records-need-a-pointer.

CURRENT STATE (updated 2026-07-15)

What this project actually is now: the home for Agent-Sudo + Constrained Autonomy (CA) — the system that removes the user's rubber-stamp permission tax by replacing approval gates with enforced deterministic constraints. The original N8N Agent Builder / N8N Builder Agent framing below is historical; those are not the active work.

Model/effort: Opus 4.8 / MEDIUM — settled 2026-07-15. Bake-off closed, Sonnet arms cancelled, per-task bake-off logging retired. Billing is hard-capped (>$20/mo impossible), so cost is not a model-selection input; Opus's real cost is more frequent Pro daily-limit hits, an explicitly accepted trade. Do not propose Sonnet; do not re-raise cost.

⚠️ TASK DESCRIPTIONS AND HANDOFFS HERE DECAY — CHECK THE ARTIFACT BEFORE DOING THE WORK. On 2026-07-15 this bit twice in one session: #149's "HAZARD: delete playbook_agent_sudo_phases" was already done the night before, and session_handoff.md was asserting a false security alarm plus a dead root token as live, into every session, for 3 days. Prune = #192; recurring guard = automation_ideas id=92. See feedback-superseding-records-need-a-pointer.

Live right now:

• Agent-Sudo is deployed and ENFORCING on both hosts (#145 closed 2026-07-14). SUDO_MD_VERIFY_ENFORCE=true, signed digest 38eca778abb0.., daemon NRestarts=0.
• CA fast-path hook v2.1 live at /opt/appdata/docker/.claude/hooks/security-enforcement.py.
• Vault: production = primary 192.168.1.88:8200. server-01 is only a CLIENT of it. The sandbox Vault on server-01 is :8201. Port 8200=prod / 8201=sandbox is the fastest tell — never infer "server-01 is involved, therefore sandbox Vault." That inversion caused a real near-miss.
• Design decisions D1–D10 + D-CB1–D-CB9 are LOCKED. The design record outranks any task title — read agent_builder/agent_sudo_design_decisions.md before re-litigating. D-CB5–D-CB9 were added 2026-07-15 (sliding per-host trip window; append-only replayed breaker log; Set B host_override; which tier-4 sources latch; trip escalation).

⚠️ P3 CODE IS COMPLETE BUT NOT DEPLOYED — BOTH HOSTS STILL RUN THE OLD sudo_rules.py. app.py + circuit_breaker.py + Set B override are written and green (85 tests OK, mutants caught) but UNDEPLOYED, so the server-01 tier-0 breaker.log hole is OPEN IN PRODUCTION until DEPLOY_RUNBOOK.md runs. Two verified deploy blockers gate it (row 187): Dockerfile never COPYs circuit_breaker.py/security/ (image would crash on boot), and docker-compose.yml has no /var/lib/agent-sudo mount (breaker resettable by docker restart — the exact D-CB6 bypass). Both were invisible to a 85/85-green suite.

Deploying is TIER-4 HUMAN-ONLY by design (D-CB7 Set B): restarting agent-sudo-daemon is refused in code, because the unit env holds SUDO_MD_VERIFY_ENFORCE — self-restart IS the gate-disarm primitive. An agent physically may not deploy this. That is the constraint model working, not a gap.

Recall corpus (2026-07-15): semantic_recall.py now indexes design docs too (DOC_ROOTS), so D-CB5–D-CB9 are searchable. context.md is deliberately excluded until #192 prunes it. MEMORY_DIR: do NOT repoint embed_memory_dir.py — its Postgres claude_memories table has no reader (verified). See row 188.

Next session, in order: #187 (Phase 0 blockers → DEPLOY_RUNBOOK.md phases 1–3; user runs the tier-4 steps) → #173 fail-open/fail-closed → #128 secrets-proxy → #147/#146 (CA P2–P5 + primary cutover 8084→8082). #205 (Vault+Bitwarden security design) is THU-first.

Then: Thu 07-16 qui/qbit memory leak (#176) + media_pipeline (#177) · Fri 07-17 clear the runway (#178) · Mon 07-20 resume business development (#179) — digital businesses deployed by Fri 07-24.

Full detail lives in the memory files, which are authoritative over this file: session_last_summary, project_near_term_schedule_jul2026, project_two_horizon_operating_model.

---

————— HISTORY BELOW THIS LINE (June 2026 →, oldest first, largely superseded) —————

Scheduled work (2026-06-16, running behind — started ~6:38 PM) — SUPERSEDED, historical

1. Vision Alignment Grill-Me — Agent Builder + N8N Builder vision + testing methodology
2. Agent Builder Agent — Deploy + Test (server-01 sandbox)
3. N8N Builder Agent — Deploy + Test (server-01 sandbox)

Architecture

• Agents run as N8N workflows on server-01 (n8n-sandbox, port 5679)
• Sandbox-first: all agents tested in sandbox before any production promotion
• server-01 sandbox stack: n8n-sandbox, postgres-sandbox, vault-sandbox, bitwarden-bridge-sandbox, vaultwarden-sandbox
• Sandbox N8N API key: prod Vault at secret/sandbox/n8n
• Sandbox reachable at 192.168.1.90

Key decisions (set during vision grill-me — 2026-06-16)

• Agent Builder Agent: builds claude_agent and script types — Ollama (llama3.1:8b) does the building, claude -p is overseer/validator
• N8N Builder Agent: builds n8n_automation types — Ollama generates workflow JSON, imports via N8N API, assigns credentials
• automation_ideas schema changes needed: rename description → task_description (full structured spec), add type (n8n_automation/claude_agent/script), add builder_status
• New agent_test_results table needed in api_business DB
• Sandbox must mirror production: AppRole, Vaultwarden, bridge all configured before any agent deploys
• Promotion = user approval required after all 4 test levels pass (not auto-promote in v1)
• Dedicated backfill session needed for all 48 existing automation_ideas rows (type + task_description)
• claude -p uses SDK credits (Pro = $20/month hard limit) — use sparingly, Ollama does the heavy lifting
• Local model: llama3.1:8b already pulled on server-01 (4.9GB, fits in RTX 2060 Super 8GB VRAM)

Testing methodology

• Four levels: Structure → Deployment → Smoke → Assertion
• LLM outputs validated on structure/side-effects only, never exact string match
• All results logged to agent_test_results table
• NTFY notification on pass and fail
• Full methodology: .claude/playbook_testing_methodology.md

Agents

Agent Builder Agent

• Status: pending — prereqs not complete
• Purpose: Receives automation spec from automation_ideas DB, uses Ollama to build claude_agent or script type automations, deploys to sandbox, runs automated tests, notifies user for promotion approval
• Builds: claude agents (via claude -p) and Python scripts (Docker containers)

N8N Builder Agent

• Status: pending — prereqs not complete
• Purpose: Receives automation spec from automation_ideas DB, uses Ollama to generate N8N workflow JSON using n8n_automations playbook as context, imports to sandbox N8N via API, assigns credentials, runs automated tests
• Will be used to build: id=12 (Media Pipeline Learning), id=7 (Friday Research Session Prep)

Related personal_projects DB rows

• id=4: N8N Workflow Builder Script (pending, weekend_block1)

Prereq checklist (must complete before any agent deployment)

• Schema: rename automation_ideas.description → task_description, add type, add builder_status, add priority
• Create agent_test_results table in api_business
• Sandbox Vault: set up AppRole auth method (credentials at /opt/appdata/docker/docker-compose/vault/approle/ on server-01)
• Sandbox Vault: store sandbox N8N API key at secret/sandbox/n8n (key name: claude-sandbox, verified working)
• Verify sandbox Bitwarden bridge ↔ Vaultwarden sandbox end-to-end (bridge on port 8080, returns [] for empty vault — correct)
• Write Agent Builder Agent playbook → .claude/playbook_agent_builder_agent.md
• Write N8N Builder Agent playbook → .claude/playbook_n8n_builder_agent.md
• Backfill session: COMPLETE. All pending rows reviewed across sessions 1-3. Blocked rows verified (32, 30, 13, 27, 23, 24, 22, 2, 21, 35, 36, 55, 56, 18 — all still blocked, no changes without proper review). Admin done: id=28 → p10, id=57 keeps p9, dummy row id=16 deleted. Session 3 (2026-06-18) final batch: ids 47, 48, 31, 26, 54.
  • id=25: type→n8n_automation, status→pending (8b model lifts hardware block), full rewrite with SDK credit model, model router + cost gate + subscription monitor, model_effort_routing_log training table
  • id=17: type→n8n_automation, expanded with token_waste_patterns DB table (evolution mechanism), JSONL-based detection, NTFY claude-audit gate, fan-out plan noted
  • id=42: expanded with hybrid trigger (Claude flags + asks), B+C output (DB + wrap-up surface), human gate permanent, hard boundary (business research = id=55 only), voice note
  • id=46: type→n8n_automation, Hermes ref removed, playbook_schema.md bootstrap design, three risk tiers, id=52 detector/id=46 executor boundary, fan-out plan noted
  • id=43: Vault AppRole required (youtube-oauth-script, new policy), vault_registry entries, port 8085 (8080 reserved for bridge), dynamic Vault IP, write verification, token refresh companion
  • id=45: hard dep on id=43 (Vault auth), niche_saturation_thresholds DB table (evolution), niche_check_results table, three-signal output (saturated/viable/unclear), id=18 stage 2 integration

New automation_ideas rows added 2026-06-18 (ids 59–67)

• id=59: Claude Audit Research Prep (n8n_automation, p12) — Saturday briefing for 12:45 PM review
• id=60: Claude Audit Topic Detector (n8n_automation, p11) — detects new topics → claude_config.audit_topics; must build first
• id=61: Claude Config Dev Work Scheduler (n8n_automation, p13) — Sunday session prep from Saturday decisions
• id=62: Monthly Security Audit (n8n_automation, p7) — first Monday of month, 7-area audit → security_audit DB
• id=63: Security Patch Backlog Handler (claude_agent, p7) — works security_audit.findings; Vault/prod always human-gate
• id=64: Business Projects Worker (claude_agent, BLOCKED p29) — 12:45–4:30 PM daily; blocked pending business_projects schema
• id=65: Personal Projects Worker (claude_agent, BLOCKED p29) — 5:30–8:00 PM daily; blocked pending personal_projects schema
• id=66: Media Pipeline Project Worker (claude_agent, BLOCKED p29) — Wednesday 5:30–8:00 PM; blocked pending id=65 + scope
• id=67: N8N Builder Agent (claude_agent, p2, ready_to_build) — builds all n8n_automation rows; deploy Monday June 22 alongside id=24

Readiness check — COMPLETE (2026-06-18 evening)

All 8 prereq checklist items verified live on server-01:

• Vault AppRole: token acquired, N8N secret readable
• N8N sandbox API: returns 200 with real key
• Bridge /items: returns [] (correct for empty sandbox vault)
• Ollama: llama3.1:8b loaded
• agent_test_results table: confirmed correct schema + FK

June 20 (Saturday) — Session progress

id=57 partially complete. Carried over from June 19 (sick day).

id=57 — DONE this session:

• Production N8N migrated from primary server → server-01 port 5678
• Vault-backed start.sh: secrets pulled from production Vault at runtime, nothing on disk
• n8n-server01 AppRole created in production Vault (scoped to secret/data/n8n read-only)
• Stale root token in Bitwarden replaced — regenerated via generate-root (3/5 keys), updated, revoked
• Traefik static route live: /traefik/dynamic/n8n.yaml → 192.168.1.90:5678
• Rollback procedure documented + all 4 checks passed: /opt/appdata/docker/docker-compose/n8n/ROLLBACK.md
• claude-policy extended: AppRole management + sys/generate-root paths added

id=57 — REMAINING (next session):

• Sandbox Vaultwarden bridge fix: 53 ciphers seeded (DB confirmed), but bw CLI WASM crash on list — bridge /items returns 500. Fix: update Vaultwarden image OR rebuild bridge with newer bw CLI OR patch bridge /items to use REST API
• N8N sandbox credentials (-sandbox suffix names) — Bridge API Key already exists; need postgres, Vault, NTFY
• Sudo bridge deploy on server-01 (Phase 2) — then disable NOPASSWD:ALL in /etc/sudoers.d/administrator on server-01

id=51 — NOT STARTED. Scope locked:

• Option B zero-exposure proxy: agent sends "run X using secret Y", proxy executes + injects secret, returns result only
• Must be live before builder agents deploy

June 20 Session 2 (continuation)

• Sandbox Vault auto-unseal deployed: vault-sandbox-unseal.sh + vault-sandbox-watch-unseal.sh + 2 systemd services, unsealed and verified
• Sandbox Vault AppRole mirror: 9 policies + 5 roles recreated from production (claude-code, claude-policy, n8n, n8n-outreach-policy, n8n-policy, n8n-rotation-policy, n8n-scheduling-policy, n8n-server01-policy, nextcloud-init)
• Production Bitwarden audit: 294 items total, 53 infra items identified (all username/password, some with notes)
• Sandbox Vaultwarden population BLOCKED: bw CLI 2026.5.0 WASM crypto error on create — use Vaultwarden REST API next session (id=70)
• sudo bridge NOT on server-01: server-01 has NOPASSWD:ALL; plan = deploy bridge first (Phase 2), then disable NOPASSWD
• ids added: 69 (Vaultwarden Version Monitor, p16), 70 (Sandbox Vaultwarden Seeder, p17)
• feedback_bw_cli_vaultwarden_create.md added to docker MEMORY.md

June 20 Session 3 (continuation)

Sandbox Vaultwarden — 53/53 dummy ciphers seeded (DB confirmed)

• Root cause of login failure: Bitwarden-Client-Version header required by Vaultwarden 1.36.0
• Original akey was encrypted with unknown master key (registration vs unlock password mismatch)
• Fix: reset sandbox@test.local password_hash + akey in postgres-sandbox via /tmp/vw_reset_and_seed.py
  • Script: derives keys from u8X_P_zlGBypRUXYq5iGIg + PBKDF2/HKDF, generates fresh user_sym_key, updates DB directly, logs in via REST API, POSTs 53 AES-256-CBC encrypted ciphers
  • Runs via: docker run --rm --network n8n-sandbox_default -v /tmp/vw_reset_and_seed.py:/tmp/script.py bitwarden-bridge:sandbox sh -c 'pip install cryptography psycopg2-binary --quiet && python3 /tmp/script.py'
  • Script at /tmp/vw_reset_and_seed.py on server-01 (also /tmp/vw_reset_and_seed.py locally)
• 53 ciphers confirmed: SELECT COUNT(*) FROM ciphers WHERE user_uuid='896b5bbb-...' → 53
• Bridge unlocks successfully with new password on every restart

BLOCKER: bw list items WASM crash (same as bw create)

• Error: "Invalid key, throwing away stored keys" × 2, then "invalid type: unit value, expected a valid string"
• Empty vault worked; non-empty vault fails — bw CLI 2026.5.0 can't decrypt Vaultwarden 1.36.0 ciphers
• Bridge /items returns 500 error; ciphers ARE in DB but not serveable via bridge
• Fix options (next session, pick one):
  1. Pull updated vaultwarden/server:latest (Docker pull currently broken due to daemon instability on server-01 — containerd-based Docker 29.x; images intact at /var/lib/containerd 26G)
  2. Rebuild bitwarden-bridge:sandbox with newer bw CLI version
  3. Patch bridge /items endpoint to use Vaultwarden REST API instead of bw CLI
• /tmp/vw_reset_and_seed.py script is reusable — if Vaultwarden is updated, just re-run to re-seed after container recreate

Docker daemon incident (unrelated to our work)

• All containers exited during session — Docker 29.x daemon crash (not caused by our actions)
• Images are in /var/lib/containerd (26G), not /var/lib/docker/overlay2 (new storage model)
• Stack brought back up via: cd /opt/appdata/docker/docker-compose/server-01 && docker compose up -d
• Compose file location confirmed: /opt/appdata/docker/docker-compose/server-01/docker-compose.yml

id=57 — COMPLETE ✅ (June 20 Session 5)

All sandbox prereqs done. Sandbox mirrors production.

Completed in Session 5:

• Bridge switched to Bitwarden cloud (megafreeman12@proton.me) — unlocks cleanly, /items returns 53
• Compose updated: BW_SERVER and NODE_TLS_REJECT_UNAUTHORIZED removed; vaultwarden-sandbox depends_on removed
• 53 dummy items seeded (dummy-infra-01 through dummy-infra-53)
• sandbox Vault secret/bitwarden updated with new master_password
• N8N sandbox credentials created: postgres-sandbox, vault-sandbox, n8n-internal-sandbox, Bridge API Key (Sandbox)
• n8n_agent_worker postgres role — api_business DB, scoped to automation_ideas + agent_test_results
• Credential in production Vault: secret/postgres/n8n-agent-worker

id=51 — DEPLOYED ON PRIMARY ✅ (June 22 session 3)

Container running healthy at https://secrets-proxy.reverseproxyserver.net on primary. Coolify service UUID: ilus0cfdkheipodw1viurg1d

June 22 session 3 facts:

• Coolify API allowed_ips was 172.16.16.1/32 — cleared to allow API access (set to empty string in instance_settings)
• Server-01 service (lv3xeu2manuimle458wbm9u9) stopped and deleted ✅
• Sandbox Vault and bridge now LAN-exposed on server-01: vault=192.168.1.90:8201, bridge=192.168.1.90:8083 (port bindings added to Coolify DB + sandbox compose file)
• secrets-proxy Vault AppRole created in prod (role: 3bf7b8a6-9e99-ca78-8e69-517881e26ea5) — creds at secret/proxy/vault-approle
• secrets-proxy Vault AppRole created in sandbox (role: f21416b3-2f3a-0461-fa38-d453a4e32865) — creds at secret/proxy/vault-approle-sandbox
• secrets_proxy DB user created in api_business (INSERT on proxy_executions only) — creds at secret/postgres/secrets-proxy
• NTFY secrets-proxy-bot created (topic: secrets-proxy-notifications, token at secret/ntfy/secrets-proxy-bot)
• Coolify bug: env vars for custom-image services can only be set via PATCH /api/v1/services/{uuid}/envs with key+value (not POST)
• Coolify DB YAML serializer mangles CMD-SHELL healthcheck single quotes (' → '') — fixed by using CMD array form in healthcheck
• psql heredoc via docker exec ... << 'SQL' doesn't work — use docker cp + psql -f instead
• pull_policy: never removed from docker-compose.yml (was server-01 workaround, no longer needed on primary)
• docker_compose_raw in Coolify DB must be updated via docker cp + psql -f for reliability

June 22 session 4 facts:

• E2E tests all passed: bridge_health_check (sandbox+prod), n8n_workflow_list (12 workflows), vault_secret_exists, vault_secret_write ✅
• proxy path format: callers pass RELATIVE path (e.g. n8n, e2e-test) — proxy prepends secret/data/ automatically
• params go inside a params: {} object in the POST body (not at top level)
• iptables DNAT rule persisted: /etc/iptables/rules.v4 saved, /etc/sysctl.d/99-local-routing.conf written ✅
  • iptables-persistent + netfilter-persistent already installed
  • Used sudo-bridge (port 8082 on 192.168.1.88) with new bash -c allowlist entries
• POST /allowlist + GET /allowlist endpoints added to sudo-bridge ✅
  • Validates tier (1/2), rejects danger-pattern matches, 409 on duplicate
  • Writes to allowlist.json on disk (daemon reloads per-request — no restart needed)
  • Audits to JSONL + Postgres, sends NTFY notification
  • New image built, pushed to gitea.local, Coolify start called via proxy shell + env_secret injection
• sudo-bridge port: 8082 on 192.168.1.88 (LAN), container name: sudo-bridge-drbjegv07256ki2lpfyr00n8
• Coolify API from within Docker network: http://172.16.16.30:8080/api/v1/ (not 172.16.16.35)
• Coolify token: vault://coolify#api_key (field name is api_key)
• To call Coolify API from proxy: use /shell with env_secrets: {"TOKEN": "vault://coolify#api_key"}
• sudo-bridge image rebuild pattern: build → push to gitea.local → stop container → POST /api/v1/services/{uuid}/start

June 22 session 4 (continued) facts:

• Shell E2E all passed: read (sandbox_exit:0), write (notify+exec), destructive (NTFY approval→exec) ✅
• proxy-sandbox-mirror image: built from /secrets-proxy/mirror/Dockerfile — must exist on HOST (proxy calls host Docker daemon via socket)
• Dockerfile fix: docker.io → docker-cli (docker.io with --no-install-recommends on Debian Trixie does not install binary)
• NTFY bug: _ntfy() was hardcoding URL to secrets-proxy-notifications regardless of topic — fixed with topic param
• NTFY ACL: secrets-proxy-bot needed write-only on secrets-proxy-approvals; Smoked5003 needed read-only — both added via ntfy CLI in container
• secrets_proxy DB user: needed SELECT on proxy_executions for ON CONFLICT DO NOTHING — granted
• Coolify token plaintext: 2|95eQySElT9uQpTXqDACWq1z9kyOaySZOZP8sBxyJaebc2bbe (Sanctum format: id|plaintext)
• Coolify "Service is already running" after stop+rm: UPDATE service_applications SET status='stopped' WHERE service_id=... then POST /start
• 524 on destructive commands: Cloudflare kills long-poll at ~100s; approval must be tapped quickly; command still runs if approved before timeout, response just lost

NEXT SESSION — in order: 0. sudo-bridge (both) — fix allowlist.json file ownership

• Problem: allowlist.json is administrator-owned on both servers — Claude can bypass POST /allowlist entirely with a direct file edit, defeating the audit log, NTFY notification, and danger-pattern veto
• Fix: chown root:root + chmod 644 allowlist.json on both primary and server-01 — daemon writes via root (systemd), Claude must use POST /allowlist endpoint
• After fix: verify POST /allowlist still works (daemon writes as root), verify direct Edit tool to allowlist.json is denied
• Sync policy decision: NO cross-bridge sync — allowlists grow organically per server (primary ≠ server-01 purposes); when a command is needed on server-01 it gets added there at that time 0a. sudo-bridge-server01 — sync POST /allowlist endpoint
• POST /allowlist was added to sudo-bridge (primary) this session but server-01 bridge container was NOT redeployed — it's still running the old image without the endpoint
• Both bridges share gitea.local/backtalk6858/sudo-bridge:latest — code is already in the image, just need to redeploy server-01 bridge (Coolify UUID: o2kz1puml1mmneyiqd96mouj) via pull + restart
• Also: server-01 allowlist.json needs the 4 new entries added today (iptables-save, 99-local-routing.conf, sysctl -p, journalctl -u) — these are primary-only entries so only add if relevant to server-01 use cases
• Verify: curl POST /allowlist and GET /allowlist on sudo-bridge-server01.reverseproxyserver.net after redeploy

1. id=51 Secrets Proxy — sandbox precondition scaffolding
   • Problem: sandbox dry-run is meaningless if the required state doesn't exist (e.g. file to delete isn't there, service isn't running)
   • Feature: before running command in proxy-sandbox-mirror, classify what the command needs to be present, check if it exists, and create/set it up if missing — so the sandbox test is a faithful replica of what would happen on the host
   • Examples: rm /tmp/foo → create /tmp/foo first; systemctl stop nginx → start nginx in sandbox first; iptables -D → add the rule first
   • Classification must handle: file existence, directory existence, running process/service, iptables rules, sysctl values
   • After scaffold → re-run command → result shown in NTFY approval body
2. id=24 Agent Builder Agent + id=67 N8N Builder Agent

• Bitwarden bridge LAN-exposed: 192.168.1.88:8083 ✅

June 21 session 2 facts for next session:

• server-01 Coolify UUID: hvzbj1gkqb5696s7cc9lcf8y
• Sandbox stack Coolify service UUID: d2celewbvh7e4fer77fcp9b5
• n8n-prod Coolify service UUID: h10eww4au274owpxozgizysh
• Traefik on primary binds to 127.0.0.1:80 only (LAN push workaround: docker save | ssh | docker load on primary then push)
• sudo_bridge DB: server_id column added to executions + allowlist_changes, default='primary'
• app.py needs: server_id='server-01' + [server-01] NTFY prefix variant for server-01 deployment
• sudo-bridge image: gitea.local/backtalk6858/sudo-bridge:latest (same image, different env vars)
• sudo-bridge-server01 Coolify UUID: o2kz1puml1mmneyiqd96mouj (port 8082, pull_policy:never)
• Vault secret created: secret/data/sudo-bridge-server01 (api_key: 443ea35b43c3e640aac3c57ed3aae06b8822ab6f)
• Host daemon running: /opt/appdata/docker/sudo-bridge/sudo_bridge_daemon.py (systemd, enabled)
• vaultwarden-sandbox REMOVED from sandbox stack (no longer needed — Bitwarden cloud dummy account used)
• app.py updated: SERVER_ID, _server_prefix(), server_id audit writes, User-Agent for Cloudflare
• NTFY approval action buttons use BRIDGE_EXTERNAL_URL=http://192.168.1.90:8082 (LAN only until Tailscale)
• NOPASSWD:ALL disabled on server-01 ✅ (June 21 session 3)
• Both bridges now on public URLs via Traefik + Cloudflare: sudo-bridge.reverseproxyserver.net + sudo-bridge-server01.reverseproxyserver.net
• Danger veto in daemon blocks rm /etc/* regardless of allowlist — /etc sudoers removal done manually
• NEXT: id=51 Secrets Execution Proxy (grill-me DONE June 21 session 4 — build next session)

June 20 Session 4 (continuation)

Docker 29.6.0 crash — diagnosed and fixed

• Root cause: nala upgrade swept in Docker 29.6.0 which has SIGSEGV null pointer dereference bug in HTTP transport during docker pull
• Fix: downgraded to 29.5.3, pinned with apt-mark hold docker-ce docker-ce-cli docker-ce-rootless-extras
• Rule: never run nala/apt upgrade without holding docker-ce first

bw CLI + Vaultwarden — permanently abandoned

• bw CLI 2026.4.1 AND 2026.5.0 both crash against Vaultwarden 1.36.0 with orgKeys null TypeError
• Root cause: Vaultwarden returns null for orgKeys in sync response for personal vaults; bw CLI expects {}
• Decision: drop Vaultwarden sandbox, use Bitwarden cloud dummy account (same as production)
• Dummy account created: megafreeman12@proton.me, master_password=Infra6746Dummy$
• All credentials stored at secret/sandbox/bitwarden in production Vault (4 fields: email, master_password, client_id, client_secret)
• Sandbox bridge Dockerfile.sandbox moved to /opt/appdata/docker/docker-compose/server-01/bitwarden-bridge/ (pinned bw CLI 2026.4.1 — moot now but kept for reference)
• bridge:sandbox image rebuilt and pushed to gitea.reverseproxyserver.net (still has bw CLI 2026.4.1)

June 22 (Monday) — Builder Agents (extended session)

Build, test, and push both builder agents to production. Work as long as it takes.

• id=24 Agent Builder Agent (claude_agent + script types)
• id=67 N8N Builder Agent (n8n_automation types)
• Prerequisites: id=51 live, id=57 FULLY complete (sandbox mirrors production)

Priority queue summary (as of 2026-06-18)

• p1: id=51 (manual), id=57 (manual) — infrastructure foundation
• p2: id=24, id=67 — builder agents (built in sessions, build everything else)
• p3: id=54 NTFY Provisioner, id=58 Human Action Gate
• p4: id=25 Cost Intelligence, id=38 Credential Emergency Rollout
• p5: id=3 Secrets Rotation, id=8 Vault Token Audit, id=52 Session Wrap-up
• p6: id=37 N8N Log Scanner, id=40 Coolify UUID Monitor
• p7: id=62 Monthly Security Audit, id=63 Security Patch Handler
• p8–p17: non-blocked pending/ready items
• p20–p29: all blocked items (builder agents skip these)

2026-06-24 Sprint Day 1 facts

• id=51 COMPLETE ✅ — sandbox precondition scaffolding deployed. _scaffold_preconditions() added to secrets-proxy app.py. Creates files/dirs, attempts service starts, scaffolds iptables -D. Silent on success, [Scaffold warning: ...] in NTFY when uncertain.
• Coolify API key: Vault secret/data/coolify → api_key (NOT a file)
• Obsidian vault PARTIAL (id=136 in_progress): /opt/appdata/obsidian/vault/ live on server-01, PARA structure, 258 memory files seeded. Remaining: ripgrep install, /recall redesign, session format Logseq→Obsidian
• Two new playbooks: playbook_secrets_proxy.md + playbook_sudo_bridge.md
• sudo-bridge rule: GET /allowlist before any POST — 409=already exists, empty exec response=rejected
• Gitea source migration logged: personal_projects id=145

Next session first task: Phase 3 — Gitea source migration (id=145). 10 services need repos. After that: Jenkins (id=141).

2026-06-24 Sprint Day 2 facts

• id=136 COMPLETE ✅ — semantic_recall.py rebuilt with ripgrep + chunked nomic-embed-text (SQLite), 290 files/434 chunks indexed; session format → Obsidian (Archives/Sessions/session-YYYY-MM-DD.md via SSH to server-01)
• Homemade skills at /opt/appdata/docker/.claude/skills/ — never search ~/.claude/plugins/
• Autonomous oversight model logged (behavior_changes id=77): AI brain grill-me + quality gates + NTFY escalation + Hermes orchestration + final human review
• sudo-bridge tier 1 NTFY: personal_projects id=146 (pending)

2026-06-24 Sprint Day 3 facts

• id=145 COMPLETE ✅ — 10 Gitea repos created (main branch), Jenkinsfiles pushed, webhooks configured
• id=141 IN PROGRESS — Jenkins deployed 192.168.1.90:8090, admin verified, JCasC working; remaining: pipeline jobs + Traefik route + E2E test
• New custom service pattern: boilerplates entry + separate Gitea repo (both always)
• dhi.io/jenkins = Docker Hardened Image, requires paid Docker subscription — not available for homelab use
• Gitea branch rename API (405) — use clone+push main+delete master pattern instead
• All 10 service repos: gitea.reverseproxyserver.net/Backtalk6858/, branch=main
• secrets-proxy PROXY_CALLERS: claude-code, agent-builder-agent, n8n-builder-agent, jenkins
• Jenkins Vault AppRole: role-id/secret-id at /opt/appdata/docker/docker-compose/jenkins/vault-approle/ on server-01
• JCasC casc.yaml lives in jenkins-home/ (volume-mounted, not in image)
• Coolify retirement plan: use for everything except Jenkins now; retire fully after Jenkins E2E passes

Next session first task: id=141 remaining — Jenkins pipeline jobs (10 via API) + Traefik route + E2E smoke test.

2026-06-24 Sprint Day 4 facts

• 10 Jenkins pipeline jobs created via API — all use gitea-credentials + gitea.local URL + GWT token=
• Traefik route LIVE: jenkins.reverseproxyserver.net → 192.168.1.90:8090 via /data/coolify/proxy/dynamic/jenkins.yaml (written via docker exec on coolify-proxy, which mounts the dir)
• Jenkins Gitea credential (gitea-credentials) added to JCasC casc.yaml (server-01 only, volume-mounted) + start.sh + docker-compose.yml — survives restarts
• gitea.local resolves inside Jenkins container (extra_hosts in docker-compose.yml)
• Coolify proxy dynamic config: /data/coolify/proxy/dynamic/ (mount: /traefik/dynamic/ inside coolify-proxy container)
• Both servers run coolify-proxy (Coolify-managed Traefik); retirement plan in project_coolify_traefik_retirement.md
• Legacy Traefik config at legacy-docker-compose/traefik/ in boilerplates — starting point for standalone Traefik when Coolify retires
• E2E BLOCKED: docker binary missing from Jenkins image despite Dockerfile having docker.io
  • Root cause: apt-get install docker.io on Debian Trixie does NOT install docker binary (known issue — package name is docker-ce or use docker-cli)
  • Fix: update Dockerfile to install docker-ce OR docker-cli, rebuild with --no-cache, push to registry, restart Jenkins

Next session first task (id=141 final steps):

1. Fix Jenkins Dockerfile: replace docker.io with docker-ce (add Docker apt repo) OR docker-cli — confirm which package provides the binary on Debian Trixie
2. Rebuild Jenkins image with --no-cache on server-01
3. Push to gitea.local/backtalk6858/jenkins:latest
4. Restart Jenkins via start.sh
5. Trigger E2E smoke test — watch push → webhook → build → docker build stage passes

2026-06-24 Sprint Day 5 facts

• id=141 COMPLETE ✅ — Full Jenkins E2E verified: Build → Push → Sandbox Deploy → Smoke Test all pass (build #9)
• Dockerfile fix: replaced docker.io with docker-ce-cli via Docker official apt repo (docker.io on Debian Trixie does NOT install docker binary); rebuilt with --no-cache
• Jenkinsfile fixes required across multiple iterations:
  • gitea-credentials (usernamePassword) used for docker login — not secrets-proxy /secret (endpoint doesn't exist)
  • Compound commands (&&, cd, $()) don't work — sudo-bridge uses subprocess.run(shlex.split()) with no shell; must use separate bridge calls
  • readJSON step not available — pipeline-utility-steps plugin not installed; replaced with python3 shell parsing
• plugins.txt: pipeline-utility-steps added (takes effect on next Jenkins image rebuild)
• server-01 allowlist.json: root:root + chmod 644 ownership (requires manual sudo to update — bootstrap problem)
  • New patterns added: docker pull , docker compose -f * up -d, docker inspect --format= *, docker ps *
• Promote to Production stage still uses compound command via secrets-proxy /shell — exits non-zero but Jenkins reports SUCCESS (fix deferred)
• Coolify API evaluation: Jenkins handles all deploy work; N8N env var rotation workflows still use Coolify API — will need migration when Coolify retires
• personal_projects id=147: coolify-traefik-retirement (p1), id=148: pam-ntfy-fingerprint-sudo (p3)
• New universal rules: feedback_evaluate_jenkins_hermes_fit.md (Jenkins/Hermes fit check on every automation design + background audit once both deployed)
• Hermes exploration day memory added: schedule full day after id=138 deploys before adding more Hermes-dependent tasks
• SSH sudo requires TTY — bridge bootstrap problem: allowlist.json is root-owned, must use manual sudo once to update it

2026-06-24 Sprint Day 6 facts

• id=147 Phase 1 COMPLETE ✅ — standalone Traefik deployed as coolify-proxy on coolify network
• Config at /opt/appdata/docker/docker-compose/traefik/ (traefik.yml + docker-compose.yml + docker-compose.test.yml)
• Mount: /data/coolify/proxy/:/traefik — certs (acme.json) preserved, dynamic configs unchanged
• All 21 routes live: 17 Docker-label + 4 file-based (jenkins, n8n, sudo-bridge-server01, coolify)
• Cloudflare Tunnel routes confirmed working end-to-end (secrets-proxy /health verified)
• Architecture clarified: Cloudflare Tunnel = encrypted transport, Traefik = HTTP-only (no TLS at origin)
  • Legacy setup was wrong: double TLS + port forwarding — current is correct
  • fileConfig.yml not needed: TLS irrelevant (Cloudflare terminates), securityHeaders not wired up anywhere
• Vault auth pattern for Claude: AppRole creds at /opt/appdata/docker/docker-compose/vault/approle/, Vault IP 172.16.16.5:8200 (coolify network), use curlimages/curl container on coolify network
• Proxy caller key: Vault secret/data/proxy/callers → claude-code field; use via docker run + Vault login, never print
• New memory rule: feedback_secrets_via_proxy_only.md — always use secrets-proxy /shell with env_secrets, never docker exec env
• server-01 Coolify proxy has zero routes — all routing handled from primary; no standalone Traefik needed on server-01
• Gitea repo for Traefik config: NOT YET CREATED — Phase 2 task

Next session first task: id=147 Phase 2 — Coolify retirement

1. Migrate N8N env var rotation workflows from Coolify API → Vault
2. Create Gitea repo for Traefik config (boilerplates entry + repo)

2026-06-24 Sprint Day 7 facts

• id=147 Phase 2 IN PROGRESS — N8N workflows migrated, Coolify stopped on primary
• N8N rotation workflow changes (committed e28a3fe in docker-compose repo):
  • DB_Password_Rotation: removed Read Vault Coolify Key + 10 Coolify PATCH/Deploy nodes; rewired ALTER → Vault Write directly
  • Admin_UI_Password_Rotation: removed Read Vault Coolify node only (coolify_key was loaded but never called Coolify API)
  • App_Token_Rotation: removed Read Vault Coolify + 19 Coolify PATCH/Deploy nodes; rewired DELETE JF Old Key → NTFY directly
  • All 3 imported to sandbox N8N (192.168.1.90:5679): IDs o8zxdYbY4Y6JRTEy, 1mM1rC9n2HjHPpJf, lDVXyar70e0w3fs9
  • Status: NOT YET TESTED in sandbox — services pick up rotated values on next Jenkins redeploy
• Coolify containers stopped on primary: coolify, coolify-realtime, coolify-redis, coolify-db, coolify-sentinel
• All 21 routes verified live post-shutdown
• New behavior rule: feedback_n8n_json_first.md — always edit workflow JSONs in git repo, import to sandbox for testing; never edit in N8N UI
• N8N playbook updated: JSON-first rule added, workflow status reset to NOT YET TESTED, secret/data/coolify removed from Vault paths
• sudo-bridge /exec endpoint confirmed (was using wrong /execute, /run); POST /exec is correct
• Vault paths no longer needed by N8N: secret/data/coolify (rotation workflows no longer call Coolify API)

Next session first task: id=147 Phase 2 — remaining steps

1. Remove coolify.yaml from /data/coolify/proxy/dynamic/ (via sudo-bridge POST /exec — needs Vault auth for bridge key)
2. Check server-01 Coolify containers (expected: just coolify-proxy which is already replaced by standalone Traefik)
3. Create Gitea repo for Traefik config + boilerplates entry
4. Verify 90-day rotation workflow dry run in sandbox with Coolify nodes removed
5. Shut down Coolify stack on primary (coolify, coolify-realtime, coolify-redis, coolify-db)
6. Shut down Coolify stack on server-01
7. Verify all routes still live after Coolify gone

2026-06-24 Sprint Day 8 facts

• id=147 COMPLETE ✅ — Coolify fully retired on both servers
  • coolify.yaml removed from /data/coolify/proxy/dynamic/ via sudo-bridge (new self-removing allowlist cleanup pattern)
  • server-01 Coolify containers stopped: coolify-sentinel + coolify-proxy
  • Gitea repo created: Backtalk6858/traefik (3 files pushed); boilerplates entry committed d2bbf8b
  • All routes verified live post-shutdown
• id=141 COMPLETE ✅ — marked complete in DB
• Phase 2 Security Hardening COMPLETE ✅:
  • id=142: bitwarden-bridge POST /items endpoint added (bw create template item → encode → create pattern)
  • id=72: secrets-proxy bitwarden:// URI field selection (bitwarden://item#field); fixed bug — /items never returns password; now uses GET /secret?item=&field=
  • id=140: zero-trust + security philosophy playbook written to memory (playbook_zero_trust_security.md)
• Phase 3 CI/CD: already complete (id=145 + id=141 done in prior days)
• end-of-session checklist: Obsidian step fixed in feedback_end_of_session_checklist.md (universal) + context-monitor.py time reminders (Logseq → Obsidian)
• Useful new pattern: self-removing sudo-bridge allowlist cleanup — write /tmp script, add tier-1 entry, execute (removes both entries), delete script
• Vault path for sudo-bridge API key: secret/data/sudo-bridge (field: first value, or api_key)
• Vault KV v2: always use secret/data/ not secret/
• bitwarden-bridge GET /items: returns id/name/username/revisionDate ONLY — never password/notes; use GET /secret?item=&field= for field reads
• secrets-proxy restart pattern: Coolify injected env vars are lost on manual restart; use /tmp startup script that reads all secrets from Vault (scratchpad: start_secrets_proxy.py)
• NotebookLLM (companion for Obsidian): NOT YET SET UP — no record in DB or memory; needs to be planned

Next session first task: Hermes deployment (id=138) — grill-me pending, read reference_open_source_ai_tools.md first

2026-06-25 Sprint Day 9 facts

• id=138 Hermes DEPLOYED ✅ — container healthy on server-01, dashboard port 9119, external route hermes.reverseproxyserver.net
• config.yaml: provider=custom, base_url=http://172.17.0.1:11434/v1, model=llama3.1:8b, tool_loop_guardrails.hard_stop_enabled=true
• Bitwarden bridge fixed: BRIDGE_API_KEY, VAULT_TOKEN, BW_CLIENTID, BW_CLIENTSECRET were empty (Coolify env var debt) — regenerated + redeployed via subprocess env injection
• Security enforcement hook v1.1 live on primary: 25 tests passing, hard-blocks sudo bypass + secret exposure
• Vault IP confirmed drifting — always resolve via docker inspect. Fixed ref 172.16.16.30 is stale.
• Jenkins = all deployments (boundary locked in memory); Hermes = all monitoring
• Hermes port 8642 already active at startup — no config change needed
• Skill files: SKILL.md format at /opt/data/skills///SKILL.md (NOT skill.yaml)
• Shell toolset for Telegram: add hermes-cli to platform_toolsets.telegram in config.yaml (PENDING)

2026-06-26 Sprint Day 10 early AM facts

• Background agent prompt discipline established after first agent looped (50+ calls/45 min) vs second agent (8 calls/4 min)
• Required rules for all background agent prompts: --max-turns 15, loop detection instruction, credential map inline, data pre-fetched, defined output format
• Automation audit: 74 items — 35 N8N / 15 Hermes / 13 Hybrid / 8 Jenkins
• required_playbooks column added to automation_ideas, 73 rows backfilled by type with Gitea raw URLs
• Both builder playbooks updated: required_playbooks fetch (Step 2) + human NTFY prompt review gate added
• id=74 Ruflo moved to personal_projects id=151; id=152 infrastructure auth map added
• Hermes user chat_id: 8871022110

2026-06-26 Sprint Day 10 afternoon facts

• Infrastructure auth map BUILT + VERIFIED: secret/data/claude/infrastructure-auth-map (3 fields: yaml/json/markdown)
• Background agent wrote it in 2 tool calls/80 seconds
• Vault IP currently 172.16.16.5 (confirm drift — always resolve dynamically)
• All 16 service paths verified present in Vault. Key field corrections vs assumptions:
  • bitwarden-bridge: field=BRIDGE_API_KEY (not api_key)
  • gitea: fields=[admin_token, admin_username]
  • bitwarden (N8N): field=bw_session
  • sudo-bridge-db exists: fields=[db, password, user]
• Auth map updated to v2 with all corrections
• Parallel agent strategy confirmed: same total tokens as serial, faster wall-clock, no added cost
• Auth map is now the canonical credential reference — eliminates agent credential-hunting problem
• New personal_projects logged: id=154 (Domain playbooks → Gitea), id=155 (required_playbooks precision pass, blocked by 154), id=156 (SessionStart hook for auth map)

2026-06-26 Sprint Day 10 afternoon session 2 facts

• id=156 COMPLETE ✅ — SessionStart hook injects full auth map from Vault at every session start (section 7 added); stale IP 172.16.16.30 fixed to dynamic resolve
• id=152 COMPLETE ✅ — marked completed in DB
• ids 157-161 LOGGED — Jenkins full deployment expansion (service-registry.json, standard Jenkinsfile template, version pinning, standard service jobs, Hermes→Jenkins trigger)
• Jenkinsfile Promote-to-Production bug FIXED — bitwarden-bridge, secrets-proxy, sudo-bridge (compound command → split calls + returncode checking)
• reference_available_tools.md CREATED — tool inventory for all design decisions; loaded at every session
• playbook_background_agent_prompts.md CREATED — mandatory wrap-up JSON + 5 discipline rules + template
• Structural wrap-up design principle added to both builder playbooks (commit 6442e7f)
• TaskCreate enforcement added to end-of-session checklist (10 tracked steps)
• credentials disambiguation added to domain vocabulary (Bitwarden=login, Vault=runtime secrets)
• Jenkins/Hermes full deployment coverage grill-me COMPLETE: N-1 versioning, boilerplates service-registry.json, Hermes→secrets-proxy→Jenkins trigger, conditional sandbox (first deploy or compose change only)
• Automation audit (74 items) now UNBLOCKED — both Jenkins + Hermes deployed
• Primary server: qbittorrent using 8.1GB RAM (abnormal — memory leak suspected), swap 7.6/8.2GB full

2026-06-26 Sprint Day 10 session 3 facts

• ids 157-160 COMPLETE ✅ — service-registry.json (33 services), standard Jenkinsfile template, 7 services version-pinned, 24 standard Jenkins jobs created (34 total)
• readJSON → readFile+JsonSlurper fix (Pipeline Utility Steps plugin not installed in Jenkins)
• proxy-caller-key mismatch: Jenkins key not in PROXY_CALLERS env var of secrets-proxy; jenkins added to proxy/callers in Vault; redeploy needed → id=163
• sudo-bridge sandbox IP binding bug: server-01 compose binds 192.168.1.88 (primary IP) → id=164
• Jellyfin Jenkins job failing pending id=163; full pipeline untested end-to-end
• Cookie jar required for Jenkins crumb across urllib requests
• Claude Max upgrade needed — daily token limits hit during background agent sessions

2026-06-26 Sprint Day 10 session 4 facts

• id=161 grill-me COMPLETE — full v1 design locked (see below)
• id=112 model decision: claude-sonnet-4-6 + --thinking medium for all claude -p automation scripts
• id=161 v2 logged as personal_projects id=165 (3-tier AI fix pipeline, next sprint)
• Phase plan locked: 11 tasks across 3 phases of background agents

id=161 v1 design (locked)

• Trigger: auto (Hermes health failure) + manual (Telegram /redeploy )
• Recovery: always full Jenkins redeploy — no restart shortcut
• Auto-fire scope: non-critical auto; critical services (secrets-proxy, authelia, sudo-bridge, bitwarden-bridge) → stop + NTFY immediately on first failure
• Max retries: 3 Jenkins attempts → if still unhealthy → stop container + NTFY user
• Hermes auth: dedicated hermes caller key in proxy/callers (new entry needed)
• Telegram format: structured /redeploy prefix
• Diagnosis: 8B model classifies failure type (constrained JSON list) before each attempt
• Logging: hermes_redeploy_log table (container_name, failure_type, hermes_diagnosis, attempt_number, triggered_by, outcome, jenkins_build_url, timestamp)
• v2 (id=165): adds Tier 1 in-place fixes + Claude -p review gate + 3-tier escalation

All 11 agent prompts written

All prompts saved to: /home/administrator/Desktop/claude/agent-builder/agent_prompts.md After /compact: read agent_prompts.md → launch Phase 1 (5 parallel) → jellyfin verify inline → Phase 2 (5 parallel) → Phase 3 (1)

Key finding this session: secrets-proxy /shell only accepts target=sandbox or target=production — server-01 ops go through sudo-bridge-server01 at http://192.168.1.90:8082 directly. Server-01 file writes: base64 python3 -c via POST /exec tier 2 (NTFY approval needed).

Phase plan for next session

Phase 1 (5 parallel agents):

1. id=163 — Redeploy secrets-proxy (BLOCKER)
2. id=164 — Fix sudo-bridge sandbox compose IP
3. Hermes config — add hermes-cli to telegram platform_toolsets
4. container-health-sweep SKILL.md
5. semantic-recall SKILL.md

[inline: jellyfin Jenkins verify after Phase 1]

Phase 2 (5 parallel agents): 6. id=161 — Hermes→Jenkins trigger v1 7. id=154 — Domain playbooks → Gitea 8. Content repurposing proposal (research + write + PDF) 9. id=135 — Update N8N playbook sandbox section 10. id=112 — Add --model claude-sonnet-4-6 --thinking medium to all claude -p automation scripts

Phase 3 (1 agent): 11. id=155 — required_playbooks precision pass (needs id=154 from Phase 2)

2026-06-27 Sprint Day 11 session 2 facts (02:40)

• Phase 2 ALL COMPLETE ✅ — all 5 agents succeeded:
  • id=161: hermes_redeploy_log table created, hermes caller key generated + stored, secrets-proxy restarted, health_trigger.py + SKILL.md written to server-01 via SSH
  • id=154: claude-playbooks Gitea repo created, 2 playbook_*.md files pushed
  • Content repurposing proposal: written to /opt/appdata/obsidian/vault/Projects/ via SCP (289 lines, 14KB)
  • id=135: N8N playbook sandbox section updated (Vaultwarden retired, Bitwarden cloud, full stack reference)
  • id=112: 3 files updated with --model claude-sonnet-4-6 --thinking medium
• Phase 3 COMPLETE ✅ — id=155: 73 rows checked, 0 broken URLs, 27 rows updated
• behavior_changes ids 2/3/4 LOGGED: python3-c sudo-bridge timeout, NTFY tier-2 notifications never sent (broken), .hermes root ownership issue
• NTFY approval priority: changed from 'high' to 'urgent' in both sudo-bridge app.py files (primary + server-01) — containers NOT yet rebuilt
• sudo-bridge-server01 Jenkins job: CREATED (Jenkinsfile.server01 in sudo-bridge Gitea repo)
• Jenkins multi-server pipeline patterns: documented in memory project_cicd_jenkins.md
• secrets-proxy Dockerfile: patched to add docker-compose-plugin — BUILD FAILING (exit code 22, Docker apt repo curl fails inside python:3.12-slim)
• behavior_changes id=5 LOGGED: secrets-proxy Dockerfile build failure
• Jenkins Dockerfile: patched on server-01 (added docker-compose-plugin to existing docker-ce-cli install)

Next session priorities (in order):

1. Fix secrets-proxy Dockerfile — switch from apt repo approach to direct binary download:
   • Replace Docker apt repo setup with: curl to GitHub releases docker-compose-linux-x86_64 → /usr/local/lib/docker/cli-plugins/docker-compose
   • Trigger secrets-proxy Jenkins job → should now build and deploy
2. Trigger sudo-bridge-server01 and sudo-bridge Jenkins jobs (now that secrets-proxy has compose plugin)
   • This deploys the urgent NTFY priority to both containers
3. Fix behavior_changes id=1: Jenkinsfile compound commands (split && into separate secrets-proxy calls in standard-redeploy.Jenkinsfile in boilerplates repo)
4. Re-run jellyfin E2E verify after id=1 fix

2026-06-27 Sprint Day 11 session 1 facts (midnight)

• Phase 1 ALL COMPLETE ✅ — all 5 agents finished:
  • id=163: secrets-proxy restarted (jenkins key active)
  • id=164: sudo-bridge compose IP fixed (192.168.1.88→192.168.1.90)
  • Hermes config: hermes-cli added to platform_toolsets.telegram, Hermes healthy
  • container-health-sweep SKILL.md: created on server-01
  • semantic-recall SKILL.md: created on server-01
• behavior_changes table CREATED in api_business DB
• Bug id=1 LOGGED: Jenkinsfile shell operators (&& and ||) incompatible with secrets-proxy shlex.split — all compound commands fail at secrets-proxy /shell
• PROXY_CALLERS format bug DISCOVERED + FIXED: app.py expects raw keys (not pre-hashed); P1-1 agent pre-hashed them causing double-hash → 403 on all auth; fixed by restart with raw keys
• Jenkins internal URL: http://192.168.1.90:8090 (direct — Cloudflare blocks automated API calls without service token)
• Jellyfin jellyfin verify: FAILED (two bugs — auth now fixed, shell operators still broken)
• agent_prompts.md FIXED: PROXY_CALLERS raw key format corrected in P1-1 and P2-1
• New anti-patterns added to playbook: PROXY_CALLERS pre-hashing, tier-2 wildcard shadowing, NTFY reliability

After /compact — next session:

1. Fix Jenkinsfile shell operator bug: split compound commands into separate secrets-proxy calls (affects all 34 Jenkins jobs using standard-redeploy.Jenkinsfile)
2. Re-run jellyfin Jenkins verify after fix
3. Launch Phase 2 (5 parallel agents): id=161, id=154, content repurposing, id=135, id=112
4. After Phase 2 id=154 completes: launch Phase 3 (id=155)

2026-07-08 Sprint Day 11 session 4 (evening) — Agent-Sudo grill-me COMPLETE + build prompts

• Agent-Sudo (id=176) grill-me DONE — D1–D10 + gap-#5 cross-cutting all locked. Design: agent_sudo_design_decisions.md. Phased build prompts P0–P5 written to agent_prompts.md (Opus 4.8 upgrades: locked-design guardrail, contract-not-brittle-script, right-sized turns, 4-level gate + agent_test_results logging, per-phase abort/rollback + idempotent pre-flight). NOT dispatched — pending user review.
• Comparison doc written: agent_sudo_vs_sudo_bridge.md (Agent-Sudo vs old bridge, side-by-side).
• Live-verified corrections: Agent-Sudo parallel port = 8084 (8083 is bitwarden-bridge; design-doc D8 was wrong); agent_test_results EXISTS in api_business (was unverified); automation_ideas/behavior_changes live in api_business (not the empty automation_ideas DB); postgres peer-auth works (no secret needed for psql).
• Coolify reality-check: Coolify is genuinely RETIRED (control plane Exited since 2026-06-25). coolify.managed=true labels on ~31 containers are VESTIGIAL static metadata, NOT active management. coolify-proxy = standalone traefik:v3.6 (name kept for cloudflared tunnels → http://coolify-proxy:80). Stale memory project_coolify_traefik_retirement.md REWRITTEN to reflect executed retirement.
• N8N caller audit: all 12 production workflows checked — ZERO call sudo-bridge. They use the now-DEAD Coolify API (172.16.16.1:8000/api/v1/deploy|services/.../envs) + Vault + bitwarden-bridge. So Agent-Sudo cutover has zero N8N surface.

⚠️ REDUNDANCY FINDING (today's decisions vs remaining sprint-day-11 work)

Remaining sprint-day-11 priorities were: (1) fix secrets-proxy Dockerfile compose-plugin, (2) trigger sudo-bridge Jenkins jobs to deploy the urgent-NTFY-priority change to both bridges, (3) fix Jenkinsfile compound-command bug (behavior_changes id=1), (4) re-run jellyfin E2E.

• #2 is now REDUNDANT — its only purpose was making the tier-2 NTFY approval notifications reliable. D4 ELIMINATES the tier-2 approval gate entirely (broken + theatrical). The old bridges are being replaced by Agent-Sudo (which has no such gate) and decommissioned in P5. Rebuilding them just to ship urgent-NTFY-priority is wasted work → DROP it. (The high→urgent app.py change stays uncommitted/undeployed; harmless.)
• #1, #3, #4 remain NECESSARY — they're secrets-proxy / Jenkins-pipeline infra, orthogonal to Agent-Sudo (Agent-Sudo replaces sudo-bridge, not secrets-proxy; the Jenkinsfile compound-command bug is a secrets-proxy /shell shlex issue, unaffected).
• New separate debt logged (not Agent-Sudo): the ~7 N8N workflows calling the dead Coolify API are silently broken → need Jenkins migration (per project_cicd_jenkins). Candidate automation_ideas / cleanup pass.

NEXT SESSION (Agent-Sudo): user reviews P0–P5 prompts → dispatch P0 (zero-risk, standalone) first, then P1→P5 in order. Switch back to Sonnet+medium (Opus 4.8/high was today-only).

Update instructions

Update at the end of every agent-builder session. Keep agent status, key decisions, and prereq checklist current.