claude-projects/agent-builder/agent_prompts.md

Background Agent Prompts — All 3 Phases

Generated: 2026-06-26 session 5

After /compact: read this file and launch Phase 1 agents first.

---

PHASE 1 — 5 parallel agents (launch first)

---

AGENT P1-1: id=163 — Redeploy secrets-proxy

You are a deployment agent running on the primary server (192.168.1.88). Restart secrets-proxy so it picks up the `jenkins` caller key that was added to Vault after the container last started.

## Task
The `jenkins` key exists in Vault at `secret/data/proxy/callers` but is not active in the running container's PROXY_CALLERS env var. Restart using the Vault-backed startup script below.

## Credentials
- AppRole role-id: /opt/appdata/docker/docker-compose/vault/approle/role-id
- AppRole secret-id: /opt/appdata/docker/docker-compose/vault/approle/secret-id
- Vault IP: resolve dynamically via docker inspect (never hardcode)

## Step-by-step

Step 1 — Resolve Vault IP:
docker inspect vault-iwaulpoi5hwirdlogshmul40 --format '{{json .NetworkSettings.Networks}}' | python3 -c "import sys,json; nets=json.load(sys.stdin); print(list(nets.values())[0]['IPAddress'])"

Step 2 — Write /tmp/restart_proxy.py with <VAULT_IP> substituted from step 1:

import hashlib, json, os, subprocess, urllib.request
VAULT_ADDR = 'http://<VAULT_IP>:8200'
ROLE_ID = open('/opt/appdata/docker/docker-compose/vault/approle/role-id').read().strip()
SECRET_ID = open('/opt/appdata/docker/docker-compose/vault/approle/secret-id').read().strip()
ENV_FILE = '/tmp/secrets-proxy.env'

def vlogin():
    body = json.dumps({'role_id': ROLE_ID, 'secret_id': SECRET_ID}).encode()
    req = urllib.request.Request(f'{VAULT_ADDR}/v1/auth/approle/login', data=body, headers={'Content-Type':'application/json'}, method='POST')
    with urllib.request.urlopen(req) as r: return json.loads(r.read())['auth']['client_token']

def vread(tok, path):
    req = urllib.request.Request(f'{VAULT_ADDR}/v1/secret/data/{path}', headers={'X-Vault-Token': tok})
    with urllib.request.urlopen(req) as r: return json.loads(r.read())['data']['data']

tok = vlogin()
callers    = vread(tok, 'proxy/callers')
approle    = vread(tok, 'proxy/vault-approle')
approle_sb = vread(tok, 'proxy/vault-approle-sandbox')
db         = vread(tok, 'postgres/secrets-proxy')
ntfy       = vread(tok, 'ntfy/secrets-proxy-bot')
urllib.request.urlopen(urllib.request.Request(f'{VAULT_ADDR}/v1/auth/token/revoke-self', data=b'{}', headers={'X-Vault-Token': tok, 'Content-Type':'application/json'}, method='POST'))


# IMPORTANT: store RAW keys — app.py hashes them internally via CALLER_KEY_TO_NAME
# Do NOT pre-hash: storing hashed keys causes double-hash, all auth fails
lines = [
    f'PROXY_CALLERS={json.dumps(callers)}',
    'VAULT_PROD_ADDR=http://192.168.1.88:8200',
    f'VAULT_PROD_ROLE_ID={approle.get("role_id","")}',
    f'VAULT_PROD_SECRET_ID={approle.get("secret_id","")}',
    'VAULT_SANDBOX_ADDR=http://192.168.1.90:8201',
    f'VAULT_SANDBOX_ROLE_ID={approle_sb.get("role_id","")}',
    f'VAULT_SANDBOX_SECRET_ID={approle_sb.get("secret_id","")}',
    'DB_HOST=172.16.16.15', 'DB_NAME=api_business',
    f'DB_USER={db.get("username","secrets_proxy")}',
    f'DB_PASSWORD={db.get("password","")}',
    'NTFY_HOST=http://ntfy-k0oooo8cckwsok80gg8kck88',
    f'NTFY_TOKEN={ntfy.get("token","")}',
    'NTFY_EXTERNAL_URL=https://secrets-proxy.reverseproxyserver.net',
    'BRIDGE_PROD_URL=http://192.168.1.88:8083',
    'BRIDGE_SANDBOX_URL=http://192.168.1.90:8083',
    'N8N_PROD_URL=http://192.168.1.90:5678',
    'N8N_SANDBOX_URL=http://192.168.1.90:5679',
    'COOLIFY_RESOURCE_UUID=ilus0cfdkheipodw1viurg1d',
    'COOLIFY_CONTAINER_NAME=secrets-proxy-ilus0cfdkheipodw1viurg1d',
]
with open(ENV_FILE, 'w') as f: f.write('\n'.join(lines) + '\n')
os.chmod(ENV_FILE, 0o600)
r = subprocess.run(['docker','compose','--env-file',ENV_FILE,'-f',
    '/opt/appdata/docker/docker-compose/secrets-proxy/docker-compose.yml',
    'up','-d','--force-recreate'], capture_output=True, text=True)
print(r.stdout[-500:] or '', r.stderr[-500:] or '')
os.remove(ENV_FILE)
print('Callers now active:', list(ch.keys()))

Step 3 — Run: python3 /tmp/restart_proxy.py

Step 4 — Wait 35 seconds, then verify:
docker ps | grep secrets-proxy
python3 -c "import urllib.request; print(urllib.request.urlopen('http://172.16.16.12:8080/health',timeout=10).read().decode())"

Step 5 — Clean up: rm -f /tmp/restart_proxy.py

## MANDATORY WRAP-UP (required regardless of success or failure)

Before stopping for ANY reason output this JSON as your final message:

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "callers now active: list them; anything else relevant"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.

---

AGENT P1-2: id=164 — Fix sudo-bridge-server01 compose IP

You are a fix agent. The sudo-bridge docker-compose.yml on server-01 has ports bound to 192.168.1.88 (primary's IP). This must be changed to 192.168.1.90 (server-01's IP) so the container survives future restarts. The container is currently running fine — fix the file only, no restart needed.

## Credentials
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40
- sudo-bridge-server01 API key: Vault path secret/data/sudo-bridge-server01, field api_key
- sudo-bridge-server01 URL: http://192.168.1.90:8082
- Auth header: Authorization: Bearer <api_key>

## Step-by-step

Write /tmp/fix_sb_ip.py and run it:

import base64, json, subprocess, urllib.request, urllib.error

# Vault login
vault_nets = json.loads(subprocess.check_output("docker inspect vault-iwaulpoi5hwirdlogshmul40 --format '{{json .NetworkSettings.Networks}}'", shell=True).decode())
VAULT_IP = list(vault_nets.values())[0]['IPAddress']
ROLE_ID = open('/opt/appdata/docker/docker-compose/vault/approle/role-id').read().strip()
SECRET_ID = open('/opt/appdata/docker/docker-compose/vault/approle/secret-id').read().strip()
body = json.dumps({'role_id': ROLE_ID, 'secret_id': SECRET_ID}).encode()
req = urllib.request.Request(f'http://{VAULT_IP}:8200/v1/auth/approle/login', data=body, headers={'Content-Type':'application/json'}, method='POST')
with urllib.request.urlopen(req) as r: tok = json.loads(r.read())['auth']['client_token']
req = urllib.request.Request(f'http://{VAULT_IP}:8200/v1/secret/data/sudo-bridge-server01', headers={'X-Vault-Token': tok})
with urllib.request.urlopen(req) as r: sb_key = json.loads(r.read())['data']['data']['api_key']
urllib.request.urlopen(urllib.request.Request(f'http://{VAULT_IP}:8200/v1/auth/token/revoke-self', data=b'{}', headers={'X-Vault-Token': tok, 'Content-Type':'application/json'}, method='POST'))

BRIDGE = 'http://192.168.1.90:8082'
HDRS = {'Content-Type':'application/json','Authorization':f'Bearer {sb_key}'}

def bexec(cmd, reason=''):
    body = json.dumps({'command': cmd, 'reason': reason}).encode()
    req = urllib.request.Request(f'{BRIDGE}/exec', data=body, headers=HDRS, method='POST')
    with urllib.request.urlopen(req, timeout=60) as r: return json.loads(r.read())

def badd(pattern, tier, reason=''):
    body = json.dumps({'pattern': pattern, 'tier': tier, 'reason': reason}).encode()
    req = urllib.request.Request(f'{BRIDGE}/allowlist', data=body, headers=HDRS, method='POST')
    try:
        with urllib.request.urlopen(req, timeout=10) as r: return json.loads(r.read())
    except urllib.error.HTTPError as e:
        if e.code == 409: return {'status': 'already_exists'}
        raise

# Find sudo-bridge container name
ps = bexec('docker ps --format {{.Names}}', 'find containers')
print('PS output:', ps)
# From output, identify the sudo-bridge container (name contains 'sudo-bridge')
# Then get its labels to find compose file path:
# bexec(f"docker inspect --format={{{{json .Config.Labels}}}} <container_name>")
# Look for com.docker.compose.project.config_files label

# Add cat to allowlist, read the compose file
badd('cat /opt/appdata/docker/docker-compose/sudo-bridge/docker-compose.yml', 1, 'read compose for IP fix')
content = bexec('cat /opt/appdata/docker/docker-compose/sudo-bridge/docker-compose.yml', 'read compose')
print('Current compose:', content)

# If file not at that path, find the actual path from docker inspect labels, then re-add to allowlist

# Fix the IP: add sed command to allowlist (tier 2 — requires NTFY approval)
compose_path = '/opt/appdata/docker/docker-compose/sudo-bridge/docker-compose.yml'  # adjust if different
badd(f"sed -i s/192.168.1.88:8082/192.168.1.90:8082/g {compose_path}", 2, 'fix IP binding in compose')
result = bexec(f"sed -i s/192.168.1.88:8082/192.168.1.90:8082/g {compose_path}", 'fix IP binding — approve in NTFY')
print('Sed result:', result)

# Verify
badd(f'cat {compose_path}', 1, 'verify IP fix')
verify = bexec(f'cat {compose_path}', 'verify fix applied')
print('After fix:', verify)

Run: python3 /tmp/fix_sb_ip.py

IMPORTANT: When you see a tier-2 command pending, the agent will block waiting for NTFY approval.
The user will tap Approve in their NTFY app. Do not retry — just wait.

## MANDATORY WRAP-UP (required regardless of success or failure)

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "compose file path that was fixed; confirm 192.168.1.90:8082 is now in the file"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.

---

AGENT P1-3: Hermes config — add hermes-cli to telegram toolset

You are a configuration agent. Add `hermes-cli` to the platform_toolsets.telegram list in Hermes config.yaml on server-01, then restart Hermes to pick up the change.

## Context
- Hermes config is at /home/administrator/.hermes/config.yaml on server-01 (volume-mounted as /opt/data inside container)
- The platform_toolsets.telegram section is a YAML list — add hermes-cli as a new entry if not already present
- Hermes docker-compose is at /opt/appdata/docker/docker-compose/hermes/docker-compose.yml on server-01

## Credentials
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40
- sudo-bridge-server01 API key: Vault path secret/data/sudo-bridge-server01, field api_key
- sudo-bridge-server01 URL: http://192.168.1.90:8082

## Step-by-step

Write /tmp/hermes_config_update.py and run it. The script should:

1. Get sudo-bridge-server01 API key from Vault (AppRole → read secret → revoke token)

2. Add cat /home/administrator/.hermes/config.yaml to allowlist (tier 1), then execute to read the config

3. Parse the YAML. Find platform_toolsets.telegram. Add hermes-cli if missing.

4. Serialize the updated YAML back to a string.

5. Base64-encode the updated config content. Then use a python3 one-liner to write it:
   - Add to allowlist (tier 2): python3 -c "import base64,builtins; builtins.open('/home/administrator/.hermes/config.yaml','w').write(base64.b64decode('<B64_CONTENT>').decode())"
   - Execute it (user approves in NTFY)

6. Restart Hermes — add to allowlist (tier 1) then execute:
   docker compose -f /opt/appdata/docker/docker-compose/hermes/docker-compose.yml up -d

7. Wait 30s, then verify health:
   docker ps | grep hermes

NOTE: sudo-bridge uses shlex.split() — no shell, no pipes, no &&. Every command is a separate POST /exec call.
NOTE: The python3 -c allowlist pattern should be: python3 -c * (wildcard to match any inline script)

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "confirm hermes-cli now in platform_toolsets.telegram; hermes container healthy"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.

---

AGENT P1-4: Create container-health-sweep SKILL.md

You are a documentation agent. Create the container-health-sweep SKILL.md on server-01 so Hermes knows how to sweep Docker containers for health issues.

## Target path on server-01
/home/administrator/.hermes/skills/homelab/container-health-sweep/SKILL.md

## Credentials
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40
- sudo-bridge-server01 API key: Vault path secret/data/sudo-bridge-server01, field api_key
- sudo-bridge-server01 URL: http://192.168.1.90:8082

## SKILL.md content (write exactly this)

---
name: container-health-sweep
description: Sweep all Docker containers and report unhealthy, stopped, or restarting ones
category: homelab
version: "1.0"
trigger:
  - scheduled
  - on_demand
---

# Container Health Sweep

Scans all Docker containers on the host and identifies those that are unhealthy, stopped, or in a restart loop.

## Command

docker ps -a --format "{{.Names}}\t{{.Status}}\t{{.Image}}"

## Health states

- HEALTHY: Up and passing healthcheck
- UNHEALTHY: Running but marked (unhealthy) in status
- STOPPED: Exited state
- RESTARTING: In restart loop (Restarting in status)

## Usage

Hermes calls this skill before triggering a Jenkins redeploy. Output is parsed to identify which containers need attention. Results logged to hermes_redeploy_log table in api_business DB.

## Escalation path

UNHEALTHY detected -> classify failure type -> trigger Jenkins redeploy (secrets-proxy /shell) -> retry up to 3x -> if still failing: docker stop + NTFY user.

Critical services (secrets-proxy, authelia, sudo-bridge, bitwarden-bridge): stop immediately on first failure, no retry.

## Step-by-step

Write /tmp/write_health_skill.py and run it:

1. Get sudo-bridge-server01 API key from Vault (AppRole login → read → revoke)

2. Create directory on server-01:
   - Add mkdir -p /home/administrator/.hermes/skills/homelab/container-health-sweep to allowlist tier 1
   - Execute it

3. Write the SKILL.md using base64 encoding:
   - Encode the SKILL.md content above as base64
   - Add to allowlist (tier 2): python3 -c * (wildcard)
   - Execute: python3 -c "import base64; open('/home/administrator/.hermes/skills/homelab/container-health-sweep/SKILL.md','w').write(base64.b64decode('<B64>').decode())"
   - User approves in NTFY

4. Verify:
   - Add cat /home/administrator/.hermes/skills/homelab/container-health-sweep/SKILL.md to allowlist tier 1
   - Execute and confirm content

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "confirm SKILL.md exists at correct path"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.

---

AGENT P1-5: Create semantic-recall SKILL.md

You are a documentation agent. Create the semantic-recall SKILL.md on server-01 so Hermes knows how to search the Obsidian vault semantically.

## Target path on server-01
/home/administrator/.hermes/skills/homelab/semantic-recall/SKILL.md

## Credentials
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40
- sudo-bridge-server01 API key: Vault path secret/data/sudo-bridge-server01, field api_key
- sudo-bridge-server01 URL: http://192.168.1.90:8082

## SKILL.md content (write exactly this)

---
name: semantic-recall
description: Semantic search over the Obsidian vault using nomic-embed-text embeddings
category: homelab
version: "1.0"
trigger:
  - on_demand
---

# Semantic Recall

Searches the Obsidian vault at /opt/appdata/obsidian/vault/ using semantic similarity. Index is a SQLite database maintained by the session-end hook.

## Command

python3 /opt/appdata/docker/.claude/scripts/semantic_recall.py query "<query string>"

## Output format

Returns top matching chunks from Obsidian notes with relevance scores. Each result includes the source file path and the matched text.

## Usage

Call before making infrastructure decisions to retrieve relevant past decisions, session notes, or playbook excerpts.

## Re-index

python3 /opt/appdata/docker/.claude/scripts/semantic_recall.py index

## Step-by-step

Same pattern as container-health-sweep SKILL.md:
1. Get sudo-bridge-server01 API key from Vault
2. mkdir -p /home/administrator/.hermes/skills/homelab/semantic-recall (allowlist tier 1)
3. Write SKILL.md via base64 python3 -c one-liner (allowlist tier 2, user approves in NTFY)
4. Verify with cat (allowlist tier 1)

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "confirm SKILL.md exists at correct path"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.

---

[INLINE after Phase 1 completes]

Trigger jellyfin Jenkins job manually and verify full pipeline succeeds end-to-end:

• Go to jenkins.reverseproxyserver.net → jellyfin job → Build Now
• Watch build log: should complete Build → Push → Sandbox Deploy → Smoke Test → Promote to Production
• If it fails, check the secrets-proxy caller key (id=163 should have fixed this)

---

PHASE 2 — 5 parallel agents (launch after Phase 1 complete + jellyfin verify)

---

AGENT P2-1: id=161 — Hermes→Jenkins trigger v1

You are a deployment agent implementing the Hermes→Jenkins health-failure trigger (v1). This is a foundational piece of infrastructure — Hermes uses this to auto-redeploy unhealthy containers via Jenkins.

## Task overview
1. Create hermes_redeploy_log table in api_business postgres
2. Generate a hermes caller key, add it to Vault proxy/callers, redeploy secrets-proxy to activate it
3. Write the health-trigger Python script to server-01 at /home/administrator/.hermes/scripts/health_trigger.py
4. Create SKILL.md for the trigger at /home/administrator/.hermes/skills/homelab/health-trigger/SKILL.md

## id=161 v1 design (implement exactly this)
- Trigger: /redeploy <service> Telegram message (manual) OR container UNHEALTHY state (auto)
- Recovery: always full Jenkins redeploy — no restart shortcut
- Critical services (secrets-proxy, authelia, sudo-bridge, bitwarden-bridge): stop immediately + NTFY on first failure, no retry
- Non-critical: 3 Jenkins redeploy attempts, then stop + NTFY if still failing
- Logging: every attempt logged to hermes_redeploy_log table
- Hermes auth: hermes caller key in secrets-proxy (new — must be generated and activated)

## Credentials
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40
- Postgres: docker exec $(docker ps --format '{{.Names}}' | grep '^postgres-') psql -U postgres -d api_business
- secrets-proxy IP: 172.16.16.12:8080 (on coolify network — use docker run --network coolify alpine/curl to reach it)
- sudo-bridge-server01 API key: Vault path secret/data/sudo-bridge-server01, field api_key
- sudo-bridge-server01 URL: http://192.168.1.90:8082

## Step 1: Create hermes_redeploy_log table

Run this SQL via docker exec postgres:

CREATE TABLE IF NOT EXISTS hermes_redeploy_log (
    id SERIAL PRIMARY KEY,
    container_name TEXT NOT NULL,
    failure_type TEXT,
    hermes_diagnosis TEXT,
    attempt_number INTEGER DEFAULT 1,
    triggered_by TEXT NOT NULL,
    outcome TEXT NOT NULL,
    jenkins_build_url TEXT,
    timestamp TIMESTAMPTZ DEFAULT NOW()
);

## Step 2: Generate and store hermes caller key

Write a Python script that:
1. Generates a secure 64-char hex key: import secrets; key = secrets.token_hex(32)
2. AppRole login to Vault
3. Read current secret/data/proxy/callers
4. Add hermes -> key to the dict
5. Write back to Vault: PUT secret/data/proxy/callers with updated dict
6. Revoke token
7. Print "hermes key generated and stored in Vault" (do NOT print the key)

## Step 3: Redeploy secrets-proxy to activate hermes key

Run the same restart_proxy.py pattern from id=163 (already in your instructions above).
The script reads proxy/callers from Vault (now includes hermes) and restarts with updated PROXY_CALLERS.
IMPORTANT: Store raw keys in PROXY_CALLERS (not pre-hashed) — app.py hashes internally. Use json.dumps(callers) not json.dumps({n: sha256(k) for n,k in callers.items()}).
Script template is identical to P1-1 — write it fresh to /tmp/restart_proxy_161.py and run it.
Verify: curl http://172.16.16.12:8080/health returns 200.

## Step 4: Write health_trigger.py to server-01

Write this script content, base64-encode it, and write to /home/administrator/.hermes/scripts/health_trigger.py via sudo-bridge-server01 tier-2 python3 -c pattern:

SCRIPT CONTENT:
#!/usr/bin/env python3
"""Hermes health trigger v1 — triggers Jenkins redeploy for unhealthy containers."""
import json, sys, subprocess, urllib.request, datetime

CRITICAL_SERVICES = ['secrets-proxy', 'authelia', 'sudo-bridge', 'bitwarden-bridge']
MAX_RETRIES = 3
# Secrets-proxy is on coolify network — reach via docker run
PROXY_URL = 'http://172.16.16.12:8080'

def get_hermes_key():
    """Read hermes caller key from Vault via AppRole."""
    import hashlib
    vault_nets = json.loads(subprocess.check_output(
        "docker inspect vault-iwaulpoi5hwirdlogshmul40 --format '{{json .NetworkSettings.Networks}}'",
        shell=True).decode())
    vault_ip = list(vault_nets.values())[0]['IPAddress']
    role_id = open('/opt/appdata/docker/docker-compose/vault/approle/role-id').read().strip()
    secret_id = open('/opt/appdata/docker/docker-compose/vault/approle/secret-id').read().strip()
    body = json.dumps({'role_id': role_id, 'secret_id': secret_id}).encode()
    req = urllib.request.Request(f'http://{vault_ip}:8200/v1/auth/approle/login',
        data=body, headers={'Content-Type':'application/json'}, method='POST')
    with urllib.request.urlopen(req) as r: tok = json.loads(r.read())['auth']['client_token']
    req = urllib.request.Request(f'http://{vault_ip}:8200/v1/secret/data/proxy/callers',
        headers={'X-Vault-Token': tok})
    with urllib.request.urlopen(req) as r: callers = json.loads(r.read())['data']['data']
    urllib.request.urlopen(urllib.request.Request(
        f'http://{vault_ip}:8200/v1/auth/token/revoke-self', data=b'{}',
        headers={'X-Vault-Token': tok, 'Content-Type':'application/json'}, method='POST'))
    return callers.get('hermes', '')

def proxy_shell(command, key, reason=''):
    body = json.dumps({'command': command, 'target': 'production', 'reason': reason}).encode()
    result = subprocess.run(
        ['docker','run','--rm','--network','coolify','curlimages/curl',
         '-sf','-X','POST','-H','Content-Type: application/json',
         '-H',f'Authorization: Bearer {key}',
         '-d',f'@-', f'{PROXY_URL}/shell'],
        input=body, capture_output=True)
    return json.loads(result.stdout) if result.stdout else {'error': result.stderr.decode()}

def trigger_jenkins(service, key):
    """Trigger Jenkins redeploy via secrets-proxy."""
    cmd = f'curl -sf http://jenkins.reverseproxyserver.net/job/{service}/build'
    return proxy_shell(cmd, key, f'Jenkins redeploy for unhealthy {service}')

def stop_container(container_name):
    subprocess.run(['docker','stop', container_name])

def ntfy_alert(message):
    subprocess.run(['docker','run','--rm','--network','coolify','curlimages/curl',
        '-sf','-X','POST','-d',message,
        'http://ntfy-k0oooo8cckwsok80gg8kck88/hermes-alerts'], capture_output=True)

def main():
    if len(sys.argv) < 2:
        print('Usage: health_trigger.py <container_name> [triggered_by]')
        sys.exit(1)
    container = sys.argv[1]
    triggered_by = sys.argv[2] if len(sys.argv) > 2 else 'auto'

    is_critical = any(svc in container for svc in CRITICAL_SERVICES)
    hermes_key = get_hermes_key()

    if is_critical:
        stop_container(container)
        ntfy_alert(f'CRITICAL: {container} stopped. Interactive session needed immediately.')
        sys.exit(0)

    for attempt in range(1, MAX_RETRIES + 1):
        trigger_jenkins(container, hermes_key)
        import time; time.sleep(60)
        status = subprocess.check_output(
            ['docker','inspect','--format','{{.State.Health.Status}}', container],
            stderr=subprocess.DEVNULL).decode().strip()
        if status == 'healthy':
            ntfy_alert(f'RECOVERED: {container} healthy after {attempt} attempt(s)')
            sys.exit(0)

    stop_container(container)
    ntfy_alert(f'FAILED: {container} still unhealthy after {MAX_RETRIES} redeploys. Stopped. Interactive session needed.')

if __name__ == '__main__':
    main()

## Step 5: Create health-trigger SKILL.md on server-01

Path: /home/administrator/.hermes/skills/homelab/health-trigger/SKILL.md

Content:
---
name: health-trigger
description: Trigger Jenkins redeploy for an unhealthy container; handles retries, critical-service fast-stop, and NTFY alerts
category: homelab
version: "1.0"
---

# Health Trigger

Invoked when Hermes detects a container is unhealthy or receives a /redeploy <service> Telegram command.

## Command

python3 /opt/data/scripts/health_trigger.py <container_name> <triggered_by>

## Examples

python3 /opt/data/scripts/health_trigger.py jellyfin auto
python3 /opt/data/scripts/health_trigger.py n8n telegram_redeploy

## Behavior

- Critical services (secrets-proxy, authelia, sudo-bridge, bitwarden-bridge): immediate stop + NTFY
- Non-critical: up to 3 Jenkins redeploy attempts, then stop + NTFY if still failing
- All attempts logged to hermes_redeploy_log table in api_business DB

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "confirm: table created, hermes key in Vault, secrets-proxy healthy, script at correct path, SKILL.md created"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.

---

AGENT P2-2: id=154 — Domain playbooks → Gitea

You are a documentation agent. Push all Claude domain playbooks from the memory directory to a dedicated Gitea repository so agents can reference them via raw URLs.

## Task
1. Check if a Gitea repo named `claude-playbooks` exists under Backtalk6858; create it if not
2. Push all .md files from /home/administrator/.claude/projects/-home-administrator-Desktop-claude/memory/ that are playbooks (filename starts with `playbook_`) to the repo
3. Verify the raw URLs work: https://gitea.reverseproxyserver.net/Backtalk6858/claude-playbooks/raw/branch/main/<filename>

## Credentials
- Gitea admin token: Vault path secret/data/gitea, field admin_token
- Gitea URL: https://gitea.reverseproxyserver.net (or gitea.local for pushes)
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40

## Step-by-step

1. AppRole login → read secret/data/gitea → get admin_token → revoke token

2. Check if repo exists:
   curl -sf -H "Authorization: token <admin_token>" https://gitea.reverseproxyserver.net/api/v1/repos/Backtalk6858/claude-playbooks

3. Create if 404:
   curl -sf -X POST -H "Authorization: token <admin_token>" -H "Content-Type: application/json" \
     -d '{"name":"claude-playbooks","description":"Claude domain playbooks for agent reference","auto_init":true,"default_branch":"main"}' \
     https://gitea.reverseproxyserver.net/api/v1/user/repos

4. Clone locally to /tmp/claude-playbooks, copy all playbook_*.md files from memory dir, commit, push

5. Verify 2-3 random raw URLs return 200

6. Note: memory dir is /home/administrator/.claude/projects/-home-administrator-Desktop-claude/memory/
   Push only files starting with playbook_ (not user_, feedback_, project_, reference_)

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "list of files pushed; repo URL"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.

---

AGENT P2-3: Content repurposing proposal

You are a research and writing agent. Research how to repurpose Claude conversation content and session notes into other formats (blog posts, LinkedIn posts, YouTube scripts, etc.). Write a structured proposal and save it as a markdown file.

## Task
1. Research content repurposing workflows — specifically for AI/homelab technical content
2. Assess what raw material exists: Obsidian session notes at /opt/appdata/obsidian/vault/Archives/Sessions/ on server-01, Claude conversation context
3. Propose a content pipeline: what types of content to create, what tools/N8N automations would be needed, estimated effort, platform targets
4. Write the proposal as a well-structured markdown document

## Output
Save the proposal to: /opt/appdata/obsidian/vault/Projects/content-repurposing-proposal.md on server-01

## Context
The user runs a technical homelab and is building AI automation infrastructure. The content has strong LinkedIn/YouTube/blog potential but currently only lives in session notes.

## Credentials
- sudo-bridge-server01 API key: Vault path secret/data/sudo-bridge-server01, field api_key (for writing to server-01)
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40
- sudo-bridge-server01 URL: http://192.168.1.90:8082

## Approach
1. Read a few recent session notes from /opt/appdata/obsidian/vault/Archives/Sessions/ on server-01 to understand the raw material (add cat of a few files to allowlist tier 1, execute)
2. Research content repurposing best practices using WebSearch if available, or reason from your knowledge
3. Write a comprehensive proposal (~800-1200 words) covering:
   - Content types (short-form, long-form, video, written)
   - Platform targets (LinkedIn, YouTube, Blog, Twitter/X)
   - Proposed N8N automation pipeline for extraction + formatting
   - Effort estimates
   - Priority recommendations
4. Write the proposal to server-01 via sudo-bridge tier-2 python3 -c base64 pattern

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "path where proposal was saved; key recommendations"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.

---

AGENT P2-4: id=135 — Update N8N playbook sandbox section

You are a documentation agent. Update the N8N playbook's sandbox section to reflect the current state: Vaultwarden is retired, Bitwarden cloud dummy account is used instead, and the current sandbox stack configuration.

## Task
Find and update the sandbox section in the N8N playbook. The playbook is either at:
- /home/administrator/.claude/projects/-home-administrator-Desktop-claude/memory/ (check for playbook_n8n_automations.md or similar)
- /opt/appdata/docker/.claude/playbooks/ on primary

## Current sandbox facts to reflect
- Vaultwarden sandbox REMOVED — Bitwarden cloud dummy account used (megafreeman12@proton.me)
- Sandbox Vault at 192.168.1.90:8201, AppRole at /opt/appdata/docker/docker-compose/server-01/vault-config/
- Sandbox N8N at 192.168.1.90:5679, API key at Vault secret/sandbox/n8n
- Bitwarden bridge sandbox at 192.168.1.90:8083
- postgres-sandbox at default postgres port (no external binding)
- sandbox Bitwarden credentials at Vault secret/sandbox/bitwarden (email, master_password, client_id, client_secret)
- N8N sandbox credentials created: postgres-sandbox, vault-sandbox, n8n-internal-sandbox, Bridge API Key (Sandbox)
- JSON-first rule: always edit workflow JSONs in git repo, import to sandbox N8N for testing; never edit directly in N8N UI
- Sandbox N8N workflow IDs for rotation workflows: o8zxdYbY4Y6JRTEy, 1mM1rC9n2HjHPpJf, lDVXyar70e0w3fs9 (NOT YET TESTED)

## Step-by-step
1. Find the N8N playbook file on primary using the Read or Bash tool
2. Read the current sandbox section
3. Update it to reflect the facts above — do not remove anything that's still accurate
4. Save the file

## Credentials
- No Vault access needed — this is a local file edit on primary

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "playbook path updated; what changed"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.

---

AGENT P2-5: id=112 — Add --model flag to claude -p scripts

You are a code update agent. Find all scripts that use `claude -p` and add `--model claude-sonnet-4-6 --thinking medium` flags to each invocation.

## Task
Search the codebase for all files containing `claude -p` invocations. For each one, add the model and thinking flags if not already present.

## Decision locked
Model: claude-sonnet-4-6
Thinking: medium
This is the hardcoded default until id=25 (Cost Intelligence System) is built.

## Search locations
- /opt/appdata/docker/ (all subdirectories)
- /home/administrator/Desktop/claude/ (all subdirectories)
- /opt/appdata/docker/.claude/scripts/
- /home/administrator/.claude/

## Step-by-step
1. Find all files containing `claude -p` using grep -r
2. For each file, read it and identify each `claude -p` invocation
3. Add --model claude-sonnet-4-6 --thinking medium if not already present
4. Preserve all other flags (--max-turns, --output-format, etc.)
5. Save each modified file

## Format
Before: claude -p "prompt text" --max-turns 15
After:  claude -p "prompt text" --model claude-sonnet-4-6 --thinking medium --max-turns 15

Before: claude -p --output-format json
After:  claude -p --model claude-sonnet-4-6 --thinking medium --output-format json

## Important
- Do not modify this prompt file itself (agent_prompts.md)
- Do not modify files in /home/administrator/.claude/projects/.../subagents/ (those are agent output logs)
- If a file already has --model flag, skip it

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["file path — what changed"],
  "actions_failed": ["file — reason"],
  "notes": "total files modified; any files skipped and why"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.

---

PHASE 3 — 1 agent (launch after Phase 2 id=154 completes)

---

AGENT P3-1: id=155 — required_playbooks precision pass

You are a database maintenance agent. The automation_ideas table has a required_playbooks column that was mass-backfilled with Gitea raw URLs by type. Now do a precision pass: verify the URLs actually resolve, and update any rows where the assigned playbooks don't match what the automation actually needs.

## Task
1. Read all rows from automation_ideas where required_playbooks IS NOT NULL
2. For each URL in each row's required_playbooks array, verify it returns 200 (HEAD request to Gitea raw URL)
3. For any URL returning 404, find the correct URL or remove the broken entry
4. For any automation whose task_description clearly references additional playbooks not in required_playbooks, add them
5. Update the DB rows where corrections are made

## Credentials
- Postgres: docker exec $(docker ps --format '{{.Names}}' | grep '^postgres-') psql -U postgres -d api_business
- Gitea token: Vault path secret/data/gitea, field admin_token (for authenticated HEAD requests)
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40

## Context
- required_playbooks is a TEXT[] column containing Gitea raw URLs
- The claude-playbooks repo should now exist at: https://gitea.reverseproxyserver.net/Backtalk6858/claude-playbooks/raw/branch/main/
- Backfill was done by automation type (n8n_automation, claude_agent, script) — some may be wrong for specific automations
- Focus on rows with status = 'pending' or 'ready_to_build' first (these are actionable)

## Step-by-step
1. AppRole → Vault → get gitea admin_token → revoke token
2. Query: SELECT id, name, type, task_description, required_playbooks FROM automation_ideas WHERE required_playbooks IS NOT NULL ORDER BY id
3. For each row, HEAD-request each URL with Gitea auth header
4. Log broken URLs, find correct paths in the claude-playbooks repo
5. UPDATE automation_ideas SET required_playbooks = ARRAY[...] WHERE id = ... for each correction
6. Report summary: rows checked, URLs fixed, rows updated

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "rows checked, URLs broken/fixed, rows updated"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.

---

---

AGENT-SUDO BUILD — Phased Prompts P0–P5

Generated: 2026-07-08 (Opus 4.8 / high). Builds projects id=176 (infrastructure_projects DB).

Design is LOCKED in agent_sudo_design_decisions.md (D1–D10 + gap-#5 cross-cutting). These prompts IMPLEMENT that design — they do not re-open it.

DO NOT launch any agent yet — this set is pending user review.

What is different from the Sonnet-era prompts above (Opus 4.8 upgrades)

1. Locked-design guardrail — each prompt opens with a "DESIGN IS LOCKED — DO NOT REDESIGN" block pointing at agent_sudo_design_decisions.md. Agents implement; they never re-decide architecture, tiers, ports, or security model.
2. Contract, not brittle script — each prompt states the contract (what must be true when done: schema, API surface, file locations, invariants) plus a reference approach. The agent writes the code to satisfy the contract instead of pasting a fragile literal script. (The Sonnet prompts embedded whole verbatim .py files that rotted.)
3. Right-sized turn budgets — per-phase, not a flat 15.
4. 4-level test gate + agent_test_results logging baked in — P2 and P4 log to agent_test_results (api_business) per the testing methodology. The Sonnet set omitted this.
5. Per-phase abort/rollback + idempotent pre-flight — each prompt has explicit pre-flight checks (safe to re-run) and an abort/rollback clause matching the gap-#5 blast-radius matrix.
6. Structural wrap-up + loop detection retained.

SHARED CONVENTIONS (referenced by every prompt below as "the shared block")

Verified infra facts (live-checked 2026-07-08 — do not re-derive; verify only if a pre-flight says to)

• Postgres (primary): container postgres-lggkk0kcgwko440kk04wowgk — resolve dynamically: docker ps --format '{{.Names}}' | grep '^postgres-'. Peer auth inside the container works: docker exec <pg> psql -U postgres -d <db> needs NO password.
• DBs: projects DB → table infrastructure_projects (Agent-Sudo = id=176). api_business DB → tables agent_test_results ✅, automation_ideas, behavior_changes. sudo_bridge DB → executions, allowlist_changes (both have server_id column, default 'primary').
• Old bridges (being replaced): primary sudo-bridge-drbjegv07256ki2lpfyr00n8 on 8082; server-01 bridge on 8082 (@192.168.1.90). Shared image gitea.local/backtalk6858/sudo-bridge:latest. API keys: Vault secret/data/sudo-bridge (primary), secret/data/sudo-bridge-server01 (server-01, field api_key).
• Agent-Sudo parallel port = 8084 (⚠️ CORRECTION to design-doc D8 which said 8083 — 8083 is taken by bitwarden-bridge; 8084/8085/8086/8087 are free on primary. Pre-flight must re-confirm free before binding.)
• Both hosts: LMDE 7 "gigi" = Debian 13 (trixie), kernel 6.12.90+deb13.1-amd64.
• Coolify is RETIRED (control plane stopped 2026-06-25). Deploy Agent-Sudo as plain docker-compose using the server-01 template (/opt/appdata/docker/non-docker-python-scripts/Docker Template/docker-compose.yml, ports bound to the host LAN IP, no coolify network, no Traefik labels) driven by Jenkins/Gitea — NEVER Coolify. coolify-proxy is a standalone Traefik (name kept for cloudflared) — leave it alone.
• Vault: container vault-iwaulpoi5hwirdlogshmul40 (IP drifts — resolve via docker inspect ... NetworkSettings.Networks). Primary AppRole creds: /opt/appdata/docker/docker-compose/vault/approle/{role-id,secret-id}. AppRole-only, orphan tokens, revoke-self after use (playbook_vault_token_rotation Step 0).

Privileged-execution path on server-01 (READ — this is the bootstrapping reality)

Agent-Sudo does not exist yet, so build agents that need root on server-01 use the existing sudo-bridge-server01 (http://192.168.1.90:8082, Bearer secret/data/sudo-bridge-server01#api_key). Rules:

• GET /allowlist before any POST /allowlist (409 = already exists). New rules via POST /allowlist only (allowlist.json is root:root — cannot be edited directly).
• sudo-bridge uses subprocess.run(shlex.split()) — NO shell: no &&, |, $(), >. Each command is a separate POST /exec. File writes = python3 -c "..." with base64 (single argv).
• ⚠️ TIER-2 NTFY APPROVAL — ROOT CAUSE CORRECTED 2026-07-14 (behavior_changes id=3, updated). Tier-2 is NOT universally broken. The NTFY server (ntfy-k0oooo8cckwsok80gg8kck88) is healthy, and server-01's bridge reaches it fine — its tier-2 approvals DO arrive on the user's phone and were approved live during P4. The real failure was the OLD PRIMARY bridge configured with an unreachable NTFY endpoint (Connection refused, topic sudo-bridge-approvals) — a dead send-path, not a dead NTFY. Rules now:
  • Tier-2 IS usable when (a) the target bridge can reach ntfy AND (b) the user is attended (interactive session / watching phone). In that case, submit the tier-2 and WAIT for approval — do NOT pre-emptively batch to manual.
  • Batch to manual-auth ONLY when the run is unattended/async, OR the target bridge's NTFY send-path is known-unreachable (currently: the old PRIMARY bridge). Then record under actions_failed as "needs manual authorization: ".
  • If a tier-2 submit returns an NTFY send failure or parks/expires, that is the misconfigured-send-path case → stop waiting, record it under actions_failed, and continue. Never stall the whole run on it.
  • Moot on primary once its Agent-Sudo daemon is live + the old bridge is decommissioned (P5).

Mandatory wrap-up (every agent, on success OR failure, as the final message)

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason (incl. any 'needs manual authorization' commands)"],
  "notes": "phase-specific confirmations + anything the next phase needs"
}

Loop guard (every agent)

If you issue the same tool call/command twice with identical arguments, STOP and emit the wrap-up with status=partially_succeeded. Respect the per-phase --max-turns.

---

P0 — DB foundation + PRE-FLIGHT AUDIT + GROUNDWORK (absorbs P1)

Blast radius: DDL = ZERO; audit = ZERO (read-only); groundwork = server-01 sandbox-side only (no production, no live data). This is the "run-while-user-does-business-research" agent: it lays every prerequisite so the focused interactive build can start at P2. It does P0's schema, a full read-only audit of all P1–P5 assumptions, and then remediates the safe/unblocking findings (= all of P1). Anything needing tier-2/manual-auth or with production blast radius is NOT done — it is batched into an explicit manual-auth list. --max-turns 40.

You are the Agent-Sudo GROUNDWORK agent, running on the primary server (192.168.1.88) with server-01 reachable at 192.168.1.90. You run while the user is doing other work. Your job: create P0's schema, audit every assumption the later build phases (P1–P5) depend on, and remediate the SAFE, unblocking findings so that when the user returns, the focused interactive build can start directly at P2 (service logic) with zero groundwork left. You do NOT build the service (P2), the security layer (P3), deploy (P4), or decommission (P5).

## DESIGN IS LOCKED — DO NOT REDESIGN
The Agent-Sudo design is fixed in agent_sudo_design_decisions.md. Create the schema and mirrors below exactly. Do not add, rename, or "improve" tiers, columns, tables, or architecture. If any contract seems wrong, record it in notes and proceed with the contract as written.

## Use the shared block (verified facts, postgres access, sudo-bridge-server01 path, tier-2-broken guard, wrap-up, loop guard).

## THE GOLDEN RULE FOR "TAKE CARE OF FINDINGS"
Remediate a finding ONLY if it is (a) read-only, or (b) server-01 sandbox-side with a clean rollback. If a fix needs a tier-2/manual-auth privileged command you cannot get as tier-1, OR would touch primary production, OR touches a live prod container — DO NOT do it. Record it verbatim under `actions_failed` as "needs manual authorization: <exact command>" and keep going. Never block on a tier-2 approval (it is broken — see shared block). A stalled agent is worse than a batched manual-auth list.

## PART A — P0 schema (do this first; zero blast radius)
Contract:
1. In the `projects` DB, table `command_audit` exists with AT LEAST these columns (types are guidance; match intent):
   - id BIGSERIAL PK
   - ts TIMESTAMPTZ DEFAULT now()
   - server_id TEXT NOT NULL           -- 'primary' | 'server-01'
   - command TEXT NOT NULL
   - decision_type TEXT NOT NULL        -- 'rule_match' | 'sandbox_test' | 'ai_eval_proposal' | 'execution' | 'circuit_breaker' | 'tier4_refuse'
   - assigned_tier SMALLINT             -- 0..4, null if refused pre-tier
   - matched_rule TEXT                  -- the SUDO.md glob/rule that matched, if any
   - host_override BOOLEAN DEFAULT false -- did a per-host override change the tier (D5)
   - evidence JSONB                     -- sandbox exit/stderr, ai-eval reasoning, etc.
   - human_verdict TEXT                 -- for ai_eval_proposal review: 'accepted'|'re_tiered'|'rejected'|null  (the GOLD training label, gap-#5 item 3)
   - exit_code INTEGER
   - duration_ms INTEGER
   - rollback_taken BOOLEAN DEFAULT false
   - verify_passed BOOLEAN              -- post-exec auto-verify result (tier 3)
   - training_signal BOOLEAN DEFAULT true  -- include this row in the local-model training export
   - Helpful indexes: (server_id, ts), (decision_type), (assigned_tier).
2. Confirm `agent_test_results` already exists in `api_business` (it does — just verify + print its columns; do NOT recreate).
3. Nothing else modified. Existing tables untouched.
Approach: write DDL as `CREATE TABLE IF NOT EXISTS ...` + `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` (idempotent), `docker cp` a .sql file into the postgres container, run via `docker exec <pg> psql -U postgres -d projects -f <file>` (NOT a heredoc). Pre-flight: `SELECT to_regclass('public.command_audit')` — if present, verify columns and only ADD missing ones; never drop/recreate.

## PART B — read-only PRE-FLIGHT AUDIT of every P1–P5 assumption (zero blast radius)
Produce a findings report (also save it to `/opt/appdata/docker/agent-sudo/preflight-audit.json` on server-01). Check and record PASS/FAIL/finding for each:
- **P1 substrate (server-01):** is `incus` installed (`which incus`)? Is root fs btrfs (`stat -f -c %T /`) → which pool backend? Do `incusbr0` / a storage pool already exist? Does `/opt/appdata/docker/agent-sudo/` exist?
- **P2 service:** is the existing sudo-bridge repo/source readable (needed as the refactor skeleton)? Read app.py + daemon + allowlist.json and note the rule count for the 0–4 remap. Is port **8084 free on BOTH hosts** (`ss -ltn` / audit; design-doc D8's 8083 is WRONG — bitwarden-bridge owns it)?
- **P3 security:** does an `agent-sudo` Vault policy/role already exist (skip-create later)? Is the Vault transit engine enabled? Is Timeshift present on server-01 (it is — confirm) and what is primary's snapshot config (record only; NEVER enable auto-restore)?
- **P4 deploy net:** is the OLD bridge on **8082 healthy on BOTH hosts** (the safety net)? Does the existing Jenkins/Gitea deploy path (Jenkinsfile.server01) exist?
- **P5:** n/a for groundwork (retention-window gated) — just note it's out of scope.

## PART C — REMEDIATE the safe/unblocking findings (= all of P1; server-01 sandbox-side only)
Fold in and execute the **P1 contract below in the "P1 — Incus + command-surface mirrors" section** in full (it is the canonical spec — follow its Contract, Reference approach, Pre-flight, and Abort/rollback exactly):
1. Capture the read-only host inventory for BOTH hosts → `/opt/appdata/docker/agent-sudo/inventory/{primary,server-01}.json` (packages, users/uids/groups, sudoers STRUCTURE only — never secret values, never file contents).
2. Install + init Incus on server-01 (btrfs pool or `dir` fallback, `incusbr0` NAT bridge) — **via sudo-bridge-server01 tier-1 if possible; if the apt install needs tier-2, defer it to the manual-auth batch and still produce everything that doesn't need it.**
3. Write `build-sandbox.sh <primary|server-01>` to `/opt/appdata/docker/agent-sudo/build-sandbox.sh`.
4. Build BOTH sandbox mirrors (`sandbox-server01`, `sandbox-primary`, both ON server-01) with the 2vCPU/2GB/10GB limits and ZERO LAN exposure (incusbr0 only). Confirm isolation.
Do NOT do anything from P2/P3/P4/P5 beyond the read-only audit in Part B.

## Abort / rollback
- Part A: abort if a DB is unreachable (touch nothing). Rollback = `DROP TABLE command_audit` (brand-new, empty). Never drop agent_test_results.
- Part C: rollback = `incus delete --force sandbox-primary sandbox-server01` + remove the btrfs pool (nothing production affected). Abort Part C (record + continue to wrap-up) if btrfs AND dir backends both fail, or if Incus install needs tier-2 you cannot get (defer via manual-auth batch — Parts A + B still stand).

## Wrap-up notes MUST include a "BUILD READINESS" section:
- command_audit columns created/verified; agent_test_results columns; row counts (0 for command_audit).
- The full pre-flight findings (path to preflight-audit.json).
- Incus version; pool backend; both containers' isolation proof (no LAN port, on incusbr0); build-sandbox.sh path.
- **A single "MANUAL-AUTH BATCH" list** = every command the user must authorize/run on return before P2 can proceed (verbatim commands), or "none — build can start at P2 immediately."
- One line: "Focused build starts at: P2" (or, if a blocker was hit, which phase and what unblocks it).

---

P1 — Incus + command-surface mirrors on server-01

NOTE (2026-07-09): P1 is now folded into the P0+GROUNDWORK agent above (Part C) so it runs while the user does business research. This section remains as the canonical contract reference that the groundwork agent follows — do NOT launch it as a separate agent unless the groundwork agent deferred it (manual-auth batch) and it needs a standalone re-run.

Blast radius: server-01 sandbox-side only (no production, no live data). --max-turns 30 (many discrete privileged steps).

You are an infrastructure agent building the sandbox substrate for Agent-Sudo on server-01 (192.168.1.90).

## DESIGN IS LOCKED — DO NOT REDESIGN (agent_sudo_design_decisions.md → D1, D7)
Container tech = Incus. Base = images:debian/13. Storage = btrfs pool (fallback: dir backend if root is not btrfs). Network = NAT via default incusbr0 (ZERO inbound LAN exposure — the sandboxes must NOT be reachable on 192.168.1.x). Two containers, BOTH on server-01: `sandbox-server01` (mirrors server-01) and `sandbox-primary` (mirrors PRIMARY's command surface, but runs on server-01 — no testing ever happens on primary). Limits each: 2 vCPU / 2GB RAM / 10GB disk. Mirror fidelity = COMMAND-SURFACE only (same OS/packages/paths/users/sudoers STRUCTURE — never secrets, never live data, never running prod containers).

## Use the shared block. Privileged server-01 ops go through sudo-bridge-server01 per the shared block's rules. HEED the tier-2-NTFY-broken warning: installing Incus needs root apt — if `apt-get install -y incus` cannot run as a tier-1 allowlist entry, DO NOT block on tier-2; record it under actions_failed as "needs manual authorization: <cmd>" and continue with whatever you can (e.g. produce the inventory + build script) so the reviewer can authorize the installs in one batch.

## Contract — what must be true when you finish (or be cleanly deferred with a manual-auth list)
1. **Read-only host inventory captured** (the mirror spec, P1 prerequisite): for BOTH hosts — `dpkg --get-selections`, users/uids + groups + sudoers STRUCTURE (never secret values), and the relevant dir tree skeleton (`/opt/appdata`, `/data/coolify`, `/etc/...` — directory structure/paths only, NOT file contents). Save each host's inventory to a file on server-01 under `/opt/appdata/docker/agent-sudo/inventory/{primary,server-01}.json`. (Primary inventory is read-only from primary; you may gather it via the shared block's primary access.)
2. **Incus installed + initialized** on server-01: btrfs storage pool created (or `dir` fallback, noted), `incusbr0` NAT bridge present, both operations idempotent.
3. **`build-sandbox.sh <primary|server-01>`** written to `/opt/appdata/docker/agent-sudo/build-sandbox.sh` — ONE parameterized script that, given a host arg, launches/refreshes the matching Incus container from images:debian/13, applies that host's package set + path skeleton + users/sudoers structure from the inventory, sets the 2vCPU/2GB/10GB limits, and attaches only incusbr0. Re-running it rebuilds the container from baseline (D1 script-rebuildable; D5 loop will later append missing deps to this script).
4. **Both containers built** and confirmed: `incus list` shows `sandbox-server01` and `sandbox-primary` RUNNING, each with NO forwarded LAN ports, each able to reach the internet (apt) but not reachable from 192.168.1.x.

## Reference approach
Prefer `incus` over legacy lxc. Gather inventories with read-only commands (dpkg, getent passwd/group, `cat /etc/sudoers.d/` STRUCTURE — pattern the rules, do not copy secrets). Generate build-sandbox.sh so it is declarative and re-runnable. Verify isolation by confirming the container's IP is on the incusbr0 subnet and no `incus config device` proxy/nic maps a LAN port.

## Pre-flight (idempotent)
- `which incus` — skip install if present. `incus storage list` / `incus network list` — skip create if present. `incus list` — if a target container exists, refresh via build-sandbox.sh rather than duplicate.
- Confirm server-01 root fs is btrfs (`stat -f -c %T /`) to decide pool backend.

## Abort / rollback
- Rollback = `incus delete --force sandbox-primary sandbox-server01` and remove the btrfs pool; nothing production is affected.
- Abort (record + stop) if: btrfs AND dir backend both fail; or Incus install needs tier-2 auth you cannot get (defer with manual-auth list).

## Wrap-up notes must confirm: incus version; pool backend chosen; both containers' isolation (no LAN port, on incusbr0); build-sandbox.sh path; any "needs manual authorization" install commands.

---

P1-HARDENING — finish + secure the sandbox substrate (2026-07-09)

Why this exists: the P0+GROUNDWORK agent built the P1 base (Incus 6.0.4, btrfs loop pool, incusbr0 NAT, both mirrors launched + isolated) but left three gaps discovered during a live networking-fix detour (see memory incus-on-docker-host-networking): (a) the sandbox internet fixes were applied live only, not baked into build-sandbox.sh — so the next rebuild wipes them; (b) the host iptables fix that lets the incus bridge forward is not reboot-persistent AND too broad (allows sandbox→LAN egress, undercutting isolation); (c) in-container provisioning was truncated by the sudo-bridge 60s exec cap and has a useradd/usermod ordering bug (adds administrator to groups before creating the user). This agent closes all three so P2 can start on a trustworthy, self-rebuilding sandbox. This is P1 hardening ONLY — do not touch P2/P3/P4/P5.

Blast radius: server-01 sandbox-side + server-01 host iptables/boot config (no production containers, no primary, no live data). --max-turns 30.

You are an infrastructure-hardening agent finishing the Agent-Sudo sandbox substrate on server-01 (192.168.1.90). The base P1 build is already done; you are closing three known gaps and proving isolation. DO NOT redesign anything and DO NOT begin P2 (service). Design is LOCKED in agent_sudo_design_decisions.md (D1, D7).

## READ FIRST (do not re-derive — these are the exact, already-diagnosed fixes)
Read the memory file `/home/administrator/.claude/projects/-home-administrator-Desktop-claude/memory/reference_incus_on_docker_host_networking.md` — it contains BOTH networking fixes verbatim (Docker FORWARD DROP → DOCKER-USER accept; broken systemd-resolved → mask + static resolv.conf) and the incus-admin access note. Use those exact commands; do not rediscover them.

## Use the shared block (verified facts, sudo-bridge-server01 path, tier-2-broken guard, wrap-up, loop guard).

## EXECUTION PATH — two lanes, pick the cheaper one per command
- **Direct SSH as `administrator@192.168.1.90` (BatchMode, key-based):** `administrator` is now in the **`incus-admin`** group, so ALL `incus ...` commands and running `build-sandbox.sh` run over plain SSH with **NO 60s cap** (the cap is a sudo-bridge limitation — this is how you avoid the truncation that broke the groundwork run). Use this lane for every incus/container/provisioning/verification step.
- **sudo-bridge-server01 (`http://192.168.1.90:8082`, tier-1 only, per shared block):** use ONLY for root-owned writes `administrator` can't do — editing `build-sandbox.sh` (root:root) and host-level iptables/systemd. Obey the no-shell / single-argv / base64-write rules. If a host iptables/systemd step is NOT tier-1-eligible, DO NOT block on tier-2 (broken) — write the artifact and DEFER the privileged enable to the MANUAL-AUTH BATCH.

## Contract — what must be true when you finish (or be cleanly deferred with a manual-auth list)
1. **DNS fix baked into `build-sandbox.sh`.** Right AFTER the `waiting for network` loop, the script must, for the launched container: `systemctl mask systemd-resolved`, stop it, and write a static `/etc/resolv.conf` (`nameserver 1.1.1.1` + the incusbr0 gateway resolver — confirm the gateway via `incus network get incusbr0 ipv4.address`). Idempotent (safe on rebuild). Verify: after a rebuild, `incus exec <ct> -- getent hosts deb.debian.org` returns IPs, exit 0 — WITHOUT any manual post-fix.
2. **Provisioning bug fixed + provisioning completes.** In `build-sandbox.sh`, reorder so every user is **created before** any `usermod -aG`/group-membership step (the current script adds `administrator` to groups before `useradd`). Make the whole provisioning block idempotent (`getent passwd X || useradd ...`, `getent group Y || groupadd ...`). Because you run via direct SSH (no 60s cap), provisioning must now run to completion. Verify: after rebuild, the mirrored users/groups/path-skeleton from the inventory JSON are all present in the container with NO ordering error in the run output.
3. **Host iptables made persistent AND tightened (close the LAN-egress hole).** The live `iptables -I DOCKER-USER -i/-o incusbr0 -j ACCEPT` rules are (a) wiped by reboot / `systemctl restart docker`, and (b) too broad — they let a sandbox reach 192.168.1.x / other RFC1918. Produce a **single boot-persistent mechanism** (a small systemd unit, e.g. `incus-sandbox-firewall.service`, `After=docker.service`, that reapplies the rules — because Docker rebuilds its chains on restart) that installs, in DOCKER-USER, this ORDER:
   - ACCEPT established/related for incusbr0,
   - from incusbr0 **DROP** to `192.168.0.0/16`, `10.0.0.0/8`, `172.16.0.0/12` **EXCEPT** the incusbr0 subnet itself (get it from `incus network get incusbr0 ipv4.address`) and the incusbr0 gateway (so container→gateway DNS still works),
   - ACCEPT incusbr0 in/out otherwise (internet egress).
   Net effect: sandbox reaches the internet (apt/DNS) but CANNOT reach the LAN. If installing/enabling the systemd unit is not tier-1-eligible via sudo-bridge, WRITE the unit file + a `restore-rules.sh` to `/opt/appdata/docker/agent-sudo/firewall/` and DEFER `systemctl enable --now incus-sandbox-firewall.service` to the MANUAL-AUTH BATCH.
4. **Both mirrors rebuilt clean from the hardened script and RE-PROVEN isolated.** Run `build-sandbox.sh server-01` then `build-sandbox.sh primary` over direct SSH. For BOTH containers confirm: RUNNING on incusbr0 with NO forwarded LAN port; `getent hosts deb.debian.org` exit 0 (internet OK); and — once the firewall rules are active — a LAN reach test FAILS (e.g. `incus exec <ct> -- ping -c1 -W2 192.168.1.88` times out / is dropped). If the firewall unit was deferred to manual-auth, run the DROP rules live to prove the isolation test, note they still need the persistent enable, and record it.

## Pre-flight (idempotent)
- `ssh -o BatchMode=yes administrator@192.168.1.90 'incus list; incus network get incusbr0 ipv4.address'` — confirm both containers + the bridge subnet/gateway.
- Check whether the DOCKER-USER incusbr0 rules are currently present (`iptables -S DOCKER-USER` via sudo-bridge) — a docker restart since 2026-07-09 may have wiped them; if gone, that's expected, the persistent unit is the fix.
- Read the current `build-sandbox.sh` first (via SSH `cat` — it's world-readable) to locate the exact `waiting for network` loop and the provisioning block before editing.

## Abort / rollback
- Rollback = `incus delete --force sandbox-primary sandbox-server01` (rebuildable) and remove any firewall unit you installed (`systemctl disable --now incus-sandbox-firewall; rm` — restores the prior live-rule state). `build-sandbox.sh` edits are in a root file — keep a `.bak` copy before editing so the change is reversible. Nothing production is touched.
- Abort (record + go to wrap-up) if: both containers can't be rebuilt; or the root write to `build-sandbox.sh` needs tier-2 you can't get (defer via manual-auth batch, still deliver the firewall artifacts).

## Wrap-up notes MUST include:
- Confirmation of each of the 4 contract items (DNS baked + verified on rebuild; provisioning ordering fixed + completes; firewall persistent-mechanism written + whether ENABLED or deferred; both mirrors rebuilt + isolation RE-PROVEN incl. the LAN-DROP test result).
- The incusbr0 subnet/gateway you used.
- A single **MANUAL-AUTH BATCH** = verbatim any privileged command the user must run (e.g. `systemctl enable --now incus-sandbox-firewall.service`), or "none — substrate is fully hardened and self-rebuilding."
- One line: "P1 substrate hardened. Next phase: P2." (or which item blocked + what unblocks it).

---

P2 — Agent-Sudo service (drop-in sudo-bridge replacement)

Blast radius: NONE while built (binds 8084; old bridge keeps 8082). --max-turns 25.

You are a service-implementation agent. Build the Agent-Sudo Python service — a DROP-IN replacement for sudo-bridge — as source in a Gitea repo. Do NOT deploy it live in this phase (that is P4).

## DESIGN IS LOCKED — DO NOT REDESIGN (agent_sudo_design_decisions.md → D4, D5, D8, D9)
Tier model 0–4 (0 auto-safe / 1 auto-reversible / 2 sandbox-first / 3 snapshot+execute / 4 hard-refuse). NO human-approval gate — the old tier-2 NTFY approve/deny is ELIMINATED, not ported. Matching engine = ordered rule list, most-specific-first (first-match-wins), fnmatch globs, regex danger-veto = auto-Tier-4, optional per-host tier override + verb class per rule. Miss behavior: primary → hard-refuse (Tier-4); server-01 → verb-heuristic default. Dumb root executor + unprivileged brain stays (separation of privilege). Response contract identical to today's bridge.

## Use the shared block.

## Contract — what must be true when you finish
1. A Gitea repo (e.g. `agent-sudo` under Backtalk6858, mirror the existing sudo-bridge repo's layout) contains:
   - `app.py` — unprivileged FastAPI brain. Endpoints preserve the OLD contract EXACTLY: `POST /exec` (Bearer) → `{exit_code, output, ...}`; `GET/POST /allowlist`; `GET /health`. REMOVE `/approve/{token}` and `/deny/{token}` and all APPROVAL_TIMEOUT logic (D4). A Tier-4 refuse returns 403 (same shape as today's allowlist-miss). Tiers 0–3 execute without any human gate.
   - `daemon` (root executor) — keep the existing dumb-root-daemon model: independent re-validation of danger-veto + allowlist before running; `subprocess.run(shlex.split())`, no shell. It must NOT trust the brain — it re-checks.
   - `SUDO.md` — the rule file: ordered globs → tier, with optional `host:` override and `verb:` class. Migrate the current allowlist.json into it per the D4 remap (old tier-1 → new 0/1 by read-vs-write; old tier-2 → new 2 or 3 by reproducible-vs-live-service; danger vetoes → 4). Seed with the researched common-command corpus (D5) — but every seeded rule with any uncertainty starts in `proposed` (inactive) state.
   - Config for `server_id` ('primary'|'server-01'), the 8084 bind, and audit → `command_audit` (projects DB) with JSONL-first→Postgres fail-soft (same resilience as today).
   - `docker-compose.yml` from the server-01 template (plain compose, host-LAN-bound, no Coolify), + `Jenkinsfile` for build→push→deploy.
2. Every decision path writes a `command_audit` row with the right `decision_type` (rule_match / execution / tier4_refuse / …) and training fields (P0 schema, gap-#5 item 3).
3. Unit-level self-check: a small `test_classify.py` that asserts representative commands land in the expected tier per host (read → 0; reversible write → 1; reproducible unknown → 2; docker/systemctl live-op → 3; danger regex → 4; primary miss → 4; server-01 miss → verb-heuristic). Log the run to `agent_test_results` (api_business) per the testing methodology (Structure + Smoke levels apply here).

## Reference approach
Start from the CURRENT sudo-bridge app.py + sudo_bridge_daemon.py as the skeleton (same repo layout, same daemon socket model, same audit resilience) and SUBTRACT the approval-gate, ADD the 0–4 tiering + per-host override + verb-heuristic + SUDO.md loader. You are refactoring a known-good service, not greenfielding. Do not paste a giant literal here — read the existing files, then write the evolved versions.

## Pre-flight (idempotent)
- Read the existing sudo-bridge repo/files first (structure + contract). Confirm the 0–4 remap against the live allowlist.json entries.
- Confirm 8084 is free on both hosts (design-doc D8 said 8083 — that is WRONG, bitwarden-bridge owns it).

## Abort / rollback
- This phase writes only to a Gitea repo + runs a local classifier test. Rollback = delete the repo/branch. Nothing runs on a host. Abort if the old bridge source can't be read (needed as the contract reference).

## Wrap-up notes must confirm: repo URL; endpoints present/removed; SUDO.md rule count (active vs proposed); test_classify pass/fail per tier; agent_test_results row id.

---

P3 — Vault AppRole + Timeshift automation + AI evaluator

Blast radius: server-01 config + additive Vault policy (existing roles untouched). --max-turns 25.

You are a security-infrastructure agent. Stand up Agent-Sudo's autonomous-security pieces (agent_sudo_design_decisions.md → D5, D9). NOTE: Bitwarden and OS-sudo-password are DROPPED from the design — do not build them.

## DESIGN IS LOCKED — DO NOT REDESIGN
Security = constrain-not-gate + execute-vs-expand split. Scoped short-TTL Vault AppRole (reuse existing rotate-and-revoke machinery; policy reads ONLY the one Agent-Sudo secret path). Signed/read-only SUDO.md via Vault transit (allowlist can't be silently widened). Scoped-undo rollback (file backup+restore / package record+remove / container via Jenkins redeploy) captured BEFORE execution; Timeshift snapshot on Tier-3; NEVER auto-restore Timeshift on primary (D3). AI evaluator produces a PROPOSED tier + audit evidence, writes the rule INACTIVE on both hosts, fires a LOW-priority NTFY (non-blocking — this is the async config review, NOT an execution gate).

## Use the shared block.

## Contract — what must be true when you finish
1. **Vault:** a new `agent-sudo` policy (read-only on exactly one secret path, e.g. `secret/data/agent-sudo`) + an AppRole bound to it, created via the rotate-and-revoke playbook (orphan tokens, revoke-self). Existing policies/roles (claude-policy, n8n-*) are provably untouched. Smoke-test: login → read the one path → confirm it CANNOT read another path → revoke.
2. **SUDO.md signing:** SUDO.md is signed with Vault transit; the daemon verifies the signature before loading (tamper → refuse to load, alert). Provide the sign + verify helpers.
3. **Scoped-undo library:** a module the service calls to capture a command-specific undo BEFORE a Tier-1/3 execution (file backup, package record, container-redeploy marker). If undo can't be captured → refuse to execute (per gap-#5 blast-radius).
4. **Timeshift automation:** take-snapshot helper for Tier-3 on server-01; on primary it may TAKE a snapshot but MUST refuse to auto-restore (hard-coded host guard).
5. **AI evaluator:** given an unknown reproducible command → route to the sandbox (P1) → analyze audit → emit a PROPOSED tier + evidence into SUDO.md (inactive) + write a `command_audit` ai_eval_proposal row + fire ONE low-priority NTFY pointing at the review conversation. It must be structurally impossible for the evaluator to mark a rule `active` (only human review does that — execute-vs-expand split).

## Reference approach
Reuse the existing AppRole tooling verbatim (playbook_vault_token_rotation Step 0). For transit, use the existing Vault transit engine. Keep the evaluator's "propose only" boundary enforced in code (no active-flag write path).

## Pre-flight (idempotent)
- Check whether the agent-sudo policy/role already exist (skip-create). Confirm Vault transit engine is enabled (enable idempotently if not).
- Confirm Timeshift present on server-01 (it is) and identify primary's snapshot config WITHOUT enabling any auto-restore.

## Abort / rollback
- Rollback = delete the agent-sudo policy + role (additive — existing auth unaffected), remove the helper scripts. Abort if AppRole login smoke-test fails, or if the least-privilege check shows the role can read a second path (security failure — do not proceed).

## Wrap-up notes must confirm: policy least-privilege proof (can read own path, cannot read another); transit sign+verify works; scoped-undo capture tested; primary auto-restore guard verified; evaluator cannot self-activate a rule.

---

P4 — Deploy both servers (parallel), run the 4-level gate

Blast radius: server-01 = the only autonomous live flip; primary flip is MANUAL/user (D8). --max-turns 20.

You are a deployment + validation agent. Deploy Agent-Sudo in PARALLEL with the old bridge and run the full 4-level test gate. server-01 FIRST; primary is user-manual.

## DESIGN IS LOCKED — DO NOT REDESIGN (agent_sudo_design_decisions.md → D8)
Order: server-01 FIRST, primary LAST. Agent-Sudo stands up on port 8084 while the old bridge keeps serving 8082 UNTOUCHED. Shadow-validate (4-level gate on 8084 + replay known-safe commands through BOTH 8082 and 8084 and diff results). Atomic swap ONLY when green: flip Agent-Sudo to 8082, demote old bridge to 8084 hot-standby (kept, not deleted), 14-day retention. server-01 swap = autonomous. **PRIMARY swap = MANUAL by the user — you must NOT flip primary; you prepare it and hand off.**

## Use the shared block.

## ⚠️ UNSIGNED INTERIM DEPLOY — READ FIRST (2026-07-10 override)
The SUDO.md transit signature is NOT active yet (Vault admin access is being resolved on a separate task; the signature is D9 tamper-evidence ONLY, not a functional dependency — the old sudo-bridge you are replacing has zero signing). Therefore deploy with signature verification OFF:
- Set **`SUDO_MD_VERIFY_ENFORCE=false`** in the Agent-Sudo container env on BOTH hosts (compose env or Jenkins-injected). With this set, the daemon runs unsigned SUDO.md exactly like the current bridge. Do NOT try to sign SUDO.md, enable transit, or touch Vault transit — that is out of scope for this phase.
- **Vault AppRole provisioning is NON-BLOCKING.** The daemon's scoped AppRole (creds at `AGENT_SUDO_APPROLE_DIR`) is only needed for signed-verify gate + secret reads. If the AppRole/creds aren't present or a Vault call 403s, log it under `actions_failed` as "needs manual: agent-sudo AppRole provisioning" and CONTINUE the deploy + gate. Agent-Sudo must come up and pass L1–L4 without it.
- **NTFY provisioning is NON-BLOCKING** (topic/token). If NTFY isn't wired, still deploy + gate; record the missing NTFY config under `actions_failed` and print gate results in the wrap-up instead. Do NOT block the deploy or swap on NTFY.
- Everything else (parallel-8084 deploy, 4-level gate, server-01 autonomous swap, primary manual hand-off) proceeds exactly as below.

## Contract — what must be true when you finish
1. **server-01:** Agent-Sudo deployed on 8084 via Jenkins/Gitea (plain compose) with `SUDO_MD_VERIFY_ENFORCE=false`. 4-level gate run and logged to `agent_test_results`:
   - L1 Structure (compose valid, container up, /health 200) · L2 Deployment (endpoints present, auth works) · L3 Smoke (a tier-0 read, a tier-1 reversible write in sandbox) · L4 Assertion (replay a set of known-safe commands through 8082 AND 8084, assert identical exit_code/output; assert a danger command → 403 tier-4; assert removed /approve endpoint → 404).
   - If ALL green → atomic swap on server-01 (8084→8082, old bridge→8084 standby). If ANY red → do NOT swap; leave 8082 serving; report.
2. **primary:** deploy Agent-Sudo on 8084 in parallel and run L1–L4 in shadow (diff against 8082) — but DO NOT swap. Produce a one-paragraph hand-off: exact manual command(s) for the user to flip primary, and the exact rollback command (flip back to old bridge on 8082).
3. NTFY on pass and fail per testing methodology.

## Reference approach
Reuse the existing Jenkins job pattern for the old bridge (Jenkinsfile.server01 already exists in the sudo-bridge repo) as the deploy mechanism, pointed at the new image. Replay set = the current allowlist.json's known-safe (read) entries — these must behave identically on both ports.

## Pre-flight (idempotent)
- Confirm 8084 free on the target host. Confirm old bridge on 8082 is healthy (the safety net) BEFORE deploying alongside. Confirm P0/P1/P2/P3 artifacts exist.

## Abort / rollback
- server-01 rollback = flip 8082 back to the old bridge (one Jenkins/compose action); Agent-Sudo returns to 8084. Old bridge retained 14 days.
- ABORT the swap on ANY L4 divergence on known-safe replay. Never touch primary's 8082. Abort if old bridge isn't healthy at start (no safety net).

## Wrap-up notes must confirm: server-01 gate results L1–L4 + swap done/blocked; primary shadow results + the exact manual swap + rollback commands for the user; agent_test_results row ids.

---

P5 — Decommission old sudo-bridge

Blast radius: removes the fallback — runs ONLY after a clean 14-day window + explicit user go. --max-turns 12.

You are a decommission agent. Retire the OLD sudo-bridge on both servers after Agent-Sudo has run clean for the retention window. DO NOT run this without an explicit user go + a clean window.

## DESIGN IS LOCKED — DO NOT REDESIGN (agent_sudo_design_decisions.md → D8; project_coolify_traefik_retirement)
Old bridge has been on 8084 hot-standby since P4's swap, 14-day retention. Coolify is RETIRED (control plane down since 2026-06-25) — there is NOTHING to deregister and nothing will resurrect a removed container. Decommission = plain docker stop + rm.

## Use the shared block.

## Contract — what must be true when you finish
1. Pre-condition PROVEN before removing anything: Agent-Sudo has been serving 8082 on the target host for ≥14 days with NO Tier-3 incident in `command_audit` and a healthy /health. If not proven → abort, remove nothing.
2. On each host (server-01 first, primary last — primary only with explicit user confirmation): `docker stop` then `docker rm` the old `sudo-bridge*` container; optionally remove its `/data/coolify/services/<uuid>/` dir; remove the old image ONLY if unused. The old bridge's restart policy is `unless-stopped` — a removed container is NOT recreated (Coolify is down), so stop+rm is permanent and clean.
3. Update projects id=176 status → completed; note the decommission in context.md.

## Pre-flight (idempotent)
- Query `command_audit` for the retention window: any circuit-breaker trips or Tier-3 failures? Confirm Agent-Sudo /health on 8082. Confirm the old bridge is the 8084 standby (not still on 8082).

## Abort / rollback
- If ANYTHING in the window looks wrong → abort, keep the old bridge. "Rollback" after removal = the retained image can be re-pulled + re-deployed to 8082 (so keep the image until the window is unambiguously clean).
- Never decommission primary without explicit user confirmation in the run request.

## Wrap-up notes must confirm: window-clean proof (incident count = 0); what was stopped/removed per host; image retained or removed; id=176 status.

---

P3-WIRING — app.py tier-2/3 + circuit-breaker (2026-07-15) — projects id=187, task #149 part B

⚠️ NAME COLLISION — READ THIS. The "P3" section ABOVE (Vault AppRole + Timeshift + AI evaluator) is the OLD P3 and is COMPLETE (task #142). P3 was redefined after its D-CB1…D-CB9 decisions were locked on 2026-07-15. THIS is the current P3. Do not run the old one.

Blast radius: ZERO. Code + tests only. No deploy, no restart, no git push. Agent-Sudo is LIVE and ENFORCING on both hosts; editing these files does not deploy them, and it must stay that way.

Human-gate protocol: deploying this code requires restarting agent-sudo-daemon, which D-CB7 classifies tier-4 / human-only on both hosts — the Set B override exists precisely to stop an agent doing it. The agent must therefore STOP at any human-only step and report it, not route around it. Resume the agent afterwards with SendMessage (its context survives).

--max-turns 25

You are wiring the LAST piece of Agent-Sudo P3: tier-2 and tier-3 execution plus the circuit-breaker into app.py. Narrow scope: `/opt/appdata/docker/docker-compose/agent-sudo/app.py` and its tests ONLY. You will NOT modify sudo_rules.py or circuit_breaker.py (both are DONE and their APIs are LOCKED). You will NOT deploy, restart, redeploy, or git push.

## THE DESIGN IS LOCKED — DO NOT REDESIGN

Read `/home/administrator/Desktop/claude/agent-builder/agent_sudo_design_decisions.md` — sections D2, D3, D5, D10 and **D-CB1 through D-CB9**. Implement exactly that. If you believe a decision is wrong, implement it as written and say so in the wrap-up `notes` — do NOT silently pick a different reading. (Two previous agents on this project each caught a real spec contradiction that way; it is the behaviour we want.)

## Context — current state, do not re-derive

- `app.py` is UNTOUCHED: zero `circuit_breaker` imports. The wiring site is ~lines 161–209, and the two capability stubs are at ~186–191:
    if d.tier == 2 and not SANDBOX_ENABLED:  -> raise HTTPException(503, 'Tier-2 sandbox pipeline not enabled on this host (requires P3)')
    if d.tier == 3 and not SNAPSHOT_ENABLED: -> raise HTTPException(503, 'Tier-3 snapshot pipeline not enabled on this host (requires P3)')
  `SANDBOX_ENABLED` / `SNAPSHOT_ENABLED` are read from env at ~lines 49–50.
- `sudo_rules.classify(command, server_id)` returns a Decision with `.tier`, `.source`, `.matched_rule`, `.host_override`, `.verb`, `.state`. **Preserve that contract.** Its three tier-4 sources are `danger_veto`, `miss_primary_refuse`, `host_override_setb`.
- `circuit_breaker.py` is DONE, 28/28 tests pass. **LOCKED API — do not rename:**
    replay(host) -> BreakerState            # ONLY source of truth; state is NEVER stored, always replayed from the log
    allows(tier, host) -> (bool, str)
    record_tier3_failure(host, cmd_id) -> BreakerState
    record_tier4_attempt(host, cmd_id) -> BreakerState   # ALWAYS attack-latches
    reset(host, actor) -> BreakerState
    classify_trip(signal) -> str            # D10 seam
  Constants: BENIGN_FAIL_THRESHOLD=3, BENIGN_WINDOW_SECONDS=900, BASE_COOLDOWN_SECONDS=600, MAX_COOLDOWN_SECONDS=14400, TRIP_ESCALATION_WINDOW_SECONDS=21600, TRIP_ESCALATION_THRESHOLD=4.
- Existing security modules already present in `agent-sudo/security/`: `timeshift.py` (raises `PrimaryRestoreForbidden`), `scoped_undo.py` (`capture_or_refuse`), `evaluator.py` (`_append_proposed_rule`, propose-only — no activation path), `sudo_sign.py`, `vault_setup.py`. READ them before wiring; do not reimplement what exists.

## D-CB8 IS THE LOAD-BEARING SUBTLETY — GET THIS RIGHT

`record_tier4_attempt()` ALWAYS attack-latches. But app.py reaches tier 4 from THREE sources, and `miss_primary_refuse` is the DEFAULT for any unlisted command. If every tier-4 latched, one typo would take the host read-only until a human reset — verified against the 2026-07-15 session, which refused every privileged command with `source=miss_primary_refuse` and would have latched on the first one and stayed latched all day.

Therefore: **ONLY `danger_veto` and `host_override_setb` call `record_tier4_attempt()`. `miss_primary_refuse` is AUDIT-ONLY — log it to command_audit, refuse the command, but do NOT touch the breaker.** The seam holds because only the CALLERS change, never the primitive.

## Contract — what must be true when you finish

1. app.py imports circuit_breaker and consults `allows(tier, host)` BEFORE executing anything. If denied, refuse with a clear error naming the breaker state + cause; do not execute.
2. Tier-2 routes through the sandbox-test path when `SANDBOX_ENABLED`, else the 503 stub stays. Tier-3 routes through the snapshot/scoped-undo path (`scoped_undo.capture_or_refuse`, `timeshift`) when `SNAPSHOT_ENABLED`, else the 503 stub stays. **The stubs must remain the behaviour when the capability is off** — this is what makes the change safe to land undeployed.
3. A tier-3 execution FAILURE calls `record_tier3_failure(host, cmd_id)`.
4. Tier-4 from `danger_veto` or `host_override_setb` calls `record_tier4_attempt(host, cmd_id)`. Tier-4 from `miss_primary_refuse` does NOT.
5. `timeshift.PrimaryRestoreForbidden` must still be honoured — never auto-restore primary.
6. Every decision + outcome is logged to `command_audit` (D2/D3: this is the local model's training data — decision_type is one of 'rule_match' | 'sandbox_test' | 'ai_eval_proposal' | 'execution' | 'circuit_breaker' | 'tier4_refuse').
7. All existing app.py behaviour for tier 0/1 is unchanged.

## Step-by-step

1. Read app.py in full, then circuit_breaker.py, sudo_rules.py, and the security/ modules. Read the design doc sections named above.
2. Wire the contract above. Match the existing code's style and error-handling idiom.
3. Write/extend `test_app.py` (create if absent; if it exists, ADD and keep all existing tests green). Must cover: breaker denies before execution; tier-3 failure records; **danger_veto latches**; **host_override_setb latches**; **miss_primary_refuse does NOT latch** (the D-CB8 test — most important); 503 stubs still returned when capabilities are off; tier 0/1 unaffected.
4. Run `cd /opt/appdata/docker/docker-compose/agent-sudo && PYTHONDONTWRITEBYTECODE=1 python3 -m unittest test_app -v` AND re-run `test_sudo_rules` + `test_circuit_breaker` to prove no regression. Paste all three verbatim summary lines in the wrap-up.
5. **Mutation-check your own suite** (both previous agents did this and it caught real gaps): break the D-CB8 branch so `miss_primary_refuse` latches, confirm the suite FAILS, restore, confirm it passes. A test that cannot fail is not evidence.
6. Write `/opt/appdata/docker/docker-compose/agent-sudo/DEPLOY_RUNBOOK.md` — the exact commands the HUMAN must run to deploy, per host, in order, with the rollback command for each and how to verify success. This is a deliverable, not an action: do NOT run any of it.

## HUMAN-GATE PROTOCOL — how to stop (this is a feature, not a failure)

Some steps are human-only BY DESIGN (D-CB7 Set B): restarting/stopping `agent-sudo-daemon`, editing its unit file, editing SUDO.md, writing breaker.log, and the primary port swap (#146). You must NOT attempt these, work around them, or stall waiting.

If you hit one, STOP IMMEDIATELY and emit the wrap-up with `status="partially_succeeded"` and populate `human_action_required`. The human will perform it and RESUME you — so `notes` must contain exactly what you were doing and what you need re-checked on resume.

## MANDATORY WRAP-UP (required regardless of success or failure)

Before stopping for ANY reason — task complete, error, human gate, or approaching turn limit — output this JSON as your final message. Do not stop without it.

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action 1 — outcome"],
  "actions_failed": ["action — reason"],
  "human_action_required": [
    {"server": "primary (192.168.1.88) | server-01 (192.168.1.90)",
     "command": "exact command to run",
     "why": "why an agent may not do this",
     "verify": "how the human confirms it worked"}
  ],
  "notes": "verbatim unittest summary lines for test_app, test_sudo_rules, test_circuit_breaker; verbatim mutant-run summary; confirmation that miss_primary_refuse does NOT latch while danger_veto and host_override_setb DO; where DEPLOY_RUNBOOK.md was written; exactly where to resume you if paused; anything ambiguous in the locked spec and what you chose"
}

`human_action_required` MUST be `[]` when empty — never omit the key.

## Constraints

- ONLY `app.py`, `test_app.py`, `DEPLOY_RUNBOOK.md`. Nothing else.
- Do NOT modify sudo_rules.py or circuit_breaker.py. Do NOT deploy, restart, redeploy, or git push.
- stdlib + the project's existing deps only.
- Do NOT create `/var/lib/agent-sudo/breaker.log` on the host.
- Use `PYTHONDONTWRITEBYTECODE=1` when running python (a `__pycache__` permission error previously masqueraded as a syntax error).

--max-turns 25
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.

---

S1-SECRETS-PROXY-SCOPE (2026-07-15) — task #128 · gameplan §4 · READY TO RUN NOW

Zero dependencies. Run this FIRST, ahead of the Agent-Sudo work — per feedback_build_the_freeing_capability_first, secrets-proxy being down taxes every future session (every secret read needs a workaround), including business-development days. This is INVESTIGATION ONLY. It produces a written scope, not a deploy.

You are scoping what remains to finish secrets-proxy. Nobody currently knows. Your job is to find out and write it down with evidence. Do not fix. Do not deploy. Do not start the container.

Verified starting facts (do not re-derive; DO verify anything you rely on further)

• Container secrets-proxy-secrets-proxy-1 = Exited (0) ~6 days ago (~2026-07-09).
• Logs show a GRACEFUL, DELIBERATE shutdown ("Application shutdown complete", "Stopping parent process [1]"). This is NOT a crash. Do not investigate it as one.
• Repo: /opt/appdata/docker/docker-compose/secrets-proxy/ — app.py, docker-compose.yml, Dockerfile, mirror/, requirements.txt. On host primary (192.168.1.88).
• Task #128 = "Finish secrets-proxy (sprint-11 #1)". Task #150 (sign proxy.md via Vault transit) is explicitly blocked on secrets-proxy being deployed.

Questions you must answer WITH EVIDENCE (quote the file/line/log/commit)

1. WHY was it stopped? Search git log, session summaries, sprint-11 notes, task #128. ⚠️ It may have been stopped for a reason that still applies. This is the most important question. If you cannot find the reason, say so explicitly — do not guess.
2. What does "sprint-11 #1" actually scope? What is incomplete in app.py / mirror/?
3. Would it start cleanly today? Determine by READING code/config, not by starting it.
4. Does docker-compose.yml carry per-host values hardcoded in a shared tracked file? (See gameplan §2 — this bug class bit twice in agent-sudo on 2026-07-15: a hardcoded port and SERVER_ID/VAULT_ADDR in a unit file. Check ports, IPs, SERVER_ID-alikes.)
5. project_coolify_env_var_debt: any ${VAR} Coolify placeholders that would bite on restart?
6. What would the proxy-only rule require to be honoured again (the /shell + vault-ref path)?

Hard constraints

• READ-ONLY. No edits, no docker compose up, no docker start, no restarts, no commits.
• NEVER surface a credential. Do not cat the env file; do not docker exec to read env; do not echo token-shaped values. To show config, print KEYS ONLY (sed 's/=.*/=<redacted>/'). The security-enforcement.py PreToolUse hook will block token patterns, and anything printed lands permanently in the transcript. If a credential is needed for a check, STOP and report.
• secrets-proxy is itself DOWN, so you cannot use it for credential reads. Do not work around this.
• If a question cannot be answered from evidence, write "UNKNOWN — could not determine because X". An honest UNKNOWN is worth more than a confident guess. Do not fill gaps by inference.

Deliverable

/opt/appdata/docker/docker-compose/secrets-proxy/SCOPE.md — findings, each with its evidence, and a proposed S2 task list with effort estimates. Flag anything human-only.

--max-turns 20

MANDATORY WRAP-UP — you cannot stop without emitting this

{"status":"succeeded|partially_succeeded|failed",
 "actions_taken":[], "actions_failed":[],
 "questions_answered":{"why_stopped":"", "sprint11_scope":"", "starts_clean":"",
                       "per_host_bug":"", "coolify_placeholders":"", "proxy_only_path":""},
 "unknowns":[], "human_action_required":[], "notes":""}

If you issue the same tool call twice with identical arguments, STOP and emit the wrap-up with status=partially_succeeded.

---

A1-TIER1-UNDO (2026-07-15) — projects id=187 gap B · gameplan §3-A1 · GATED

🔴 DO NOT RUN THIS UNTIL THE HUMAN HAS DECIDED (a) vs (b) BELOW. The decision is the user's, not yours. If dispatched without a recorded decision, STOP immediately and emit the wrap-up with human_action_required: ["A1 decision (a) or (b)"].

The defect (verified 2026-07-15 19:55)

• security/scoped_undo.py:186 → REQUIRES_UNDO_TIERS = (1, 3)
• app.py:244 → scoped_undo.capture_or_refuse(command, 3, server_id=SERVER_ID) — tier hardcoded to 3. Nothing ever captures undo for tier 1.
• Result: tier-1 commands auto-execute with NO undo capture, contradicting D4 ("tier 1 = auto-execute + capture scoped-undo").

The conflict a human must resolve first

The P3 task contract said "tier 0/1 behaviour unchanged"; D4 says tier 1 captures undo. The P3 agent followed the contract, pinned it with a test, and flagged it — correctly.

• (a) Implement D4: tier 1 captures scoped-undo. Cost: undo-capture latency on every tier-1 command (tier 1 is the volume tier). Benefit: tier 1 becomes reversible.
• (b) Amend D4: drop 1 from REQUIRES_UNDO_TIERS; tier 1 is fire-and-forget. Cheaper, but tier 1 becomes irreversible.

If (a) — your task

1. Make the tier argument at app.py:244 reflect the ACTUAL tier, not a hardcoded 3.
2. Ensure tier-1 flows through capture_or_refuse per REQUIRES_UNDO_TIERS.
3. ⚠️ The existing test pins the WRONG behaviour (tier 1 = no undo capture). You MUST update it, or you will lock in the bug. Find it, understand why the P3 agent wrote it, then change it.
4. Decide + implement what happens when tier-1 undo capture FAILS. Note the precedent in gap C: a failed PRE-exec snapshot refuses without calling record_tier3_failure(), because counting infrastructure faults toward the benign trip threshold would let broken infra disarm the tier. Apply the SAME reasoning to tier 1 and say so explicitly in your notes. ⚠️ Refusing every tier-1 command when undo capture is broken would stall the user's whole workflow — that is the exact stall CA-D7 forbids. Think about this; flag it, do not guess.
5. All 85 existing tests must still pass. Add tests for the new behaviour.

If (b) — your task

Drop 1 from REQUIRES_UNDO_TIERS, update D4 in agent_sudo_design_decisions.md to say tier 1 is deliberately irreversible AND why, keep the existing test, add a comment at app.py:244 explaining the hardcoded 3 is now correct-by-decision.

Hard constraints

• Permitted files ONLY: app.py, security/scoped_undo.py, test_app.py, agent_sudo_design_decisions.md (path b only). Anything else → flag, do not touch. (On 2026-07-15 the P3 agent's two most valuable findings were blockers OUTSIDE its permitted set, which it flagged rather than fixed. That is the system working. Do the same.)
• NEVER restart/stop agent-sudo-daemon, edit its unit, edit SUDO.md, or write /var/lib/agent-sudo/breaker.log — all tier-4 human-only by design (D-CB7).
• NEVER fire a danger-veto or host_override_setb command on a live host — it latches a production breaker. Both hosts are LIVE as of 2026-07-15 (server-01 :8082, primary :8084).
• Run tests with PYTHONDONTWRITEBYTECODE=1 python3 -m unittest test_app test_sudo_rules test_circuit_breaker (baseline: Ran 85 tests ... OK).
• A green suite proves less than you think. On 2026-07-15, 85/85 passed while the image crashed on boot and the breaker was resettable by docker restart. If you change behaviour, prove it with a test that FAILS before your change and passes after. Say which test that is.

--max-turns 25

MANDATORY WRAP-UP — you cannot stop without emitting this

{"status":"succeeded|partially_succeeded|failed|blocked_on_decision",
 "decision_taken":"a|b|none",
 "actions_taken":[], "actions_failed":[],
 "tests":{"baseline_85_pass":true, "new_tests_added":[], "test_that_fails_before_change":""},
 "undo_capture_failure_behaviour":"", "files_touched":[], "files_flagged_not_touched":[],
 "human_action_required":[], "notes":""}

---

A7-SPEC-GAPS-AC (2026-07-15) — projects id=187 gaps A + C · gameplan §3-A7 · small

Confirm-and-close two flagged spec gaps. Analysis + recommendation; change only gap A, and only if the human has said which way.

Gap A: ATTACK_TIER4_SOURCES = ('danger_veto', 'host_override_setb'). But classify() also reaches tier 4 via rule_match (explicit SUDO.md tier-4 rule) and miss_verb_heuristic (server-01). Both are currently audit-only (no latch). Question for the human: should an explicit tier-4 SUDO.md rule be an attack signal? One-line change to ATTACK_TIER4_SOURCES. ⚠️ D-CB8 deliberately EXCLUDES miss_primary_refuse — it is the DEFAULT for unlisted commands, so latching on it would take a host read-only on the first typo. Do NOT "fix" that. If you propose adding it, you have misunderstood the design.

Gap C: A failed PRE-exec snapshot refuses 503 without calling record_tier3_failure() — the command never ran, and counting infrastructure faults toward the benign trip threshold would let a broken timeshift disarm tier 3. Verify this reasoning against the code and confirm or refute.

Permitted files: app.py (gap A one-liner only, only once decided). Everything else: report only. Do not fire tier-4 commands on live hosts. All 85 tests must pass.

--max-turns 12

MANDATORY WRAP-UP

{"status":"succeeded|partially_succeeded|failed",
 "gap_a":{"recommendation":"", "changed":false, "reasoning":""},
 "gap_c":{"confirmed":true, "reasoning":""},
 "actions_taken":[], "actions_failed":[], "human_action_required":[], "notes":""}