claude-projects/agent-builder/agent_prompts.md

# Background Agent Prompts — All 3 Phases
# Generated: 2026-06-26 session 5
# After /compact: read this file and launch Phase 1 agents first.

---

## PHASE 1 — 5 parallel agents (launch first)

---

### AGENT P1-1: id=163 — Redeploy secrets-proxy

```
You are a deployment agent running on the primary server (192.168.1.88). Restart secrets-proxy so it picks up the `jenkins` caller key that was added to Vault after the container last started.

## Task
The `jenkins` key exists in Vault at `secret/data/proxy/callers` but is not active in the running container's PROXY_CALLERS env var. Restart using the Vault-backed startup script below.

## Credentials
- AppRole role-id: /opt/appdata/docker/docker-compose/vault/approle/role-id
- AppRole secret-id: /opt/appdata/docker/docker-compose/vault/approle/secret-id
- Vault IP: resolve dynamically via docker inspect (never hardcode)

## Step-by-step

Step 1 — Resolve Vault IP:
docker inspect vault-iwaulpoi5hwirdlogshmul40 --format '{{json .NetworkSettings.Networks}}' | python3 -c "import sys,json; nets=json.load(sys.stdin); print(list(nets.values())[0]['IPAddress'])"

Step 2 — Write /tmp/restart_proxy.py with <VAULT_IP> substituted from step 1:

import hashlib, json, os, subprocess, urllib.request
VAULT_ADDR = 'http://<VAULT_IP>:8200'
ROLE_ID = open('/opt/appdata/docker/docker-compose/vault/approle/role-id').read().strip()
SECRET_ID = open('/opt/appdata/docker/docker-compose/vault/approle/secret-id').read().strip()
ENV_FILE = '/tmp/secrets-proxy.env'

def vlogin():
    body = json.dumps({'role_id': ROLE_ID, 'secret_id': SECRET_ID}).encode()
    req = urllib.request.Request(f'{VAULT_ADDR}/v1/auth/approle/login', data=body, headers={'Content-Type':'application/json'}, method='POST')
    with urllib.request.urlopen(req) as r: return json.loads(r.read())['auth']['client_token']

def vread(tok, path):
    req = urllib.request.Request(f'{VAULT_ADDR}/v1/secret/data/{path}', headers={'X-Vault-Token': tok})
    with urllib.request.urlopen(req) as r: return json.loads(r.read())['data']['data']

tok = vlogin()
callers    = vread(tok, 'proxy/callers')
approle    = vread(tok, 'proxy/vault-approle')
approle_sb = vread(tok, 'proxy/vault-approle-sandbox')
db         = vread(tok, 'postgres/secrets-proxy')
ntfy       = vread(tok, 'ntfy/secrets-proxy-bot')
urllib.request.urlopen(urllib.request.Request(f'{VAULT_ADDR}/v1/auth/token/revoke-self', data=b'{}', headers={'X-Vault-Token': tok, 'Content-Type':'application/json'}, method='POST'))

ch = {n: hashlib.sha256(k.encode()).hexdigest() for n, k in callers.items()}
lines = [
    f'PROXY_CALLERS={json.dumps(ch)}',
    'VAULT_PROD_ADDR=http://192.168.1.88:8200',
    f'VAULT_PROD_ROLE_ID={approle.get("role_id","")}',
    f'VAULT_PROD_SECRET_ID={approle.get("secret_id","")}',
    'VAULT_SANDBOX_ADDR=http://192.168.1.90:8201',
    f'VAULT_SANDBOX_ROLE_ID={approle_sb.get("role_id","")}',
    f'VAULT_SANDBOX_SECRET_ID={approle_sb.get("secret_id","")}',
    'DB_HOST=172.16.16.15', 'DB_NAME=api_business',
    f'DB_USER={db.get("username","secrets_proxy")}',
    f'DB_PASSWORD={db.get("password","")}',
    'NTFY_HOST=http://ntfy-k0oooo8cckwsok80gg8kck88',
    f'NTFY_TOKEN={ntfy.get("token","")}',
    'NTFY_EXTERNAL_URL=https://secrets-proxy.reverseproxyserver.net',
    'BRIDGE_PROD_URL=http://192.168.1.88:8083',
    'BRIDGE_SANDBOX_URL=http://192.168.1.90:8083',
    'N8N_PROD_URL=http://192.168.1.90:5678',
    'N8N_SANDBOX_URL=http://192.168.1.90:5679',
    'COOLIFY_RESOURCE_UUID=ilus0cfdkheipodw1viurg1d',
    'COOLIFY_CONTAINER_NAME=secrets-proxy-ilus0cfdkheipodw1viurg1d',
]
with open(ENV_FILE, 'w') as f: f.write('\n'.join(lines) + '\n')
os.chmod(ENV_FILE, 0o600)
r = subprocess.run(['docker','compose','--env-file',ENV_FILE,'-f',
    '/opt/appdata/docker/docker-compose/secrets-proxy/docker-compose.yml',
    'up','-d','--force-recreate'], capture_output=True, text=True)
print(r.stdout[-500:] or '', r.stderr[-500:] or '')
os.remove(ENV_FILE)
print('Callers now active:', list(ch.keys()))

Step 3 — Run: python3 /tmp/restart_proxy.py

Step 4 — Wait 35 seconds, then verify:
docker ps | grep secrets-proxy
python3 -c "import urllib.request; print(urllib.request.urlopen('http://172.16.16.12:8080/health',timeout=10).read().decode())"

Step 5 — Clean up: rm -f /tmp/restart_proxy.py

## MANDATORY WRAP-UP (required regardless of success or failure)

Before stopping for ANY reason output this JSON as your final message:

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "callers now active: list them; anything else relevant"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.
```

---

### AGENT P1-2: id=164 — Fix sudo-bridge-server01 compose IP

```
You are a fix agent. The sudo-bridge docker-compose.yml on server-01 has ports bound to 192.168.1.88 (primary's IP). This must be changed to 192.168.1.90 (server-01's IP) so the container survives future restarts. The container is currently running fine — fix the file only, no restart needed.

## Credentials
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40
- sudo-bridge-server01 API key: Vault path secret/data/sudo-bridge-server01, field api_key
- sudo-bridge-server01 URL: http://192.168.1.90:8082
- Auth header: Authorization: Bearer <api_key>

## Step-by-step

Write /tmp/fix_sb_ip.py and run it:

import base64, json, subprocess, urllib.request, urllib.error

# Vault login
vault_nets = json.loads(subprocess.check_output("docker inspect vault-iwaulpoi5hwirdlogshmul40 --format '{{json .NetworkSettings.Networks}}'", shell=True).decode())
VAULT_IP = list(vault_nets.values())[0]['IPAddress']
ROLE_ID = open('/opt/appdata/docker/docker-compose/vault/approle/role-id').read().strip()
SECRET_ID = open('/opt/appdata/docker/docker-compose/vault/approle/secret-id').read().strip()
body = json.dumps({'role_id': ROLE_ID, 'secret_id': SECRET_ID}).encode()
req = urllib.request.Request(f'http://{VAULT_IP}:8200/v1/auth/approle/login', data=body, headers={'Content-Type':'application/json'}, method='POST')
with urllib.request.urlopen(req) as r: tok = json.loads(r.read())['auth']['client_token']
req = urllib.request.Request(f'http://{VAULT_IP}:8200/v1/secret/data/sudo-bridge-server01', headers={'X-Vault-Token': tok})
with urllib.request.urlopen(req) as r: sb_key = json.loads(r.read())['data']['data']['api_key']
urllib.request.urlopen(urllib.request.Request(f'http://{VAULT_IP}:8200/v1/auth/token/revoke-self', data=b'{}', headers={'X-Vault-Token': tok, 'Content-Type':'application/json'}, method='POST'))

BRIDGE = 'http://192.168.1.90:8082'
HDRS = {'Content-Type':'application/json','Authorization':f'Bearer {sb_key}'}

def bexec(cmd, reason=''):
    body = json.dumps({'command': cmd, 'reason': reason}).encode()
    req = urllib.request.Request(f'{BRIDGE}/exec', data=body, headers=HDRS, method='POST')
    with urllib.request.urlopen(req, timeout=60) as r: return json.loads(r.read())

def badd(pattern, tier, reason=''):
    body = json.dumps({'pattern': pattern, 'tier': tier, 'reason': reason}).encode()
    req = urllib.request.Request(f'{BRIDGE}/allowlist', data=body, headers=HDRS, method='POST')
    try:
        with urllib.request.urlopen(req, timeout=10) as r: return json.loads(r.read())
    except urllib.error.HTTPError as e:
        if e.code == 409: return {'status': 'already_exists'}
        raise

# Find sudo-bridge container name
ps = bexec('docker ps --format {{.Names}}', 'find containers')
print('PS output:', ps)
# From output, identify the sudo-bridge container (name contains 'sudo-bridge')
# Then get its labels to find compose file path:
# bexec(f"docker inspect --format={{{{json .Config.Labels}}}} <container_name>")
# Look for com.docker.compose.project.config_files label

# Add cat to allowlist, read the compose file
badd('cat /opt/appdata/docker/docker-compose/sudo-bridge/docker-compose.yml', 1, 'read compose for IP fix')
content = bexec('cat /opt/appdata/docker/docker-compose/sudo-bridge/docker-compose.yml', 'read compose')
print('Current compose:', content)

# If file not at that path, find the actual path from docker inspect labels, then re-add to allowlist

# Fix the IP: add sed command to allowlist (tier 2 — requires NTFY approval)
compose_path = '/opt/appdata/docker/docker-compose/sudo-bridge/docker-compose.yml'  # adjust if different
badd(f"sed -i s/192.168.1.88:8082/192.168.1.90:8082/g {compose_path}", 2, 'fix IP binding in compose')
result = bexec(f"sed -i s/192.168.1.88:8082/192.168.1.90:8082/g {compose_path}", 'fix IP binding — approve in NTFY')
print('Sed result:', result)

# Verify
badd(f'cat {compose_path}', 1, 'verify IP fix')
verify = bexec(f'cat {compose_path}', 'verify fix applied')
print('After fix:', verify)

Run: python3 /tmp/fix_sb_ip.py

IMPORTANT: When you see a tier-2 command pending, the agent will block waiting for NTFY approval.
The user will tap Approve in their NTFY app. Do not retry — just wait.

## MANDATORY WRAP-UP (required regardless of success or failure)

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "compose file path that was fixed; confirm 192.168.1.90:8082 is now in the file"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.
```

---

### AGENT P1-3: Hermes config — add hermes-cli to telegram toolset

```
You are a configuration agent. Add `hermes-cli` to the platform_toolsets.telegram list in Hermes config.yaml on server-01, then restart Hermes to pick up the change.

## Context
- Hermes config is at /home/administrator/.hermes/config.yaml on server-01 (volume-mounted as /opt/data inside container)
- The platform_toolsets.telegram section is a YAML list — add hermes-cli as a new entry if not already present
- Hermes docker-compose is at /opt/appdata/docker/docker-compose/hermes/docker-compose.yml on server-01

## Credentials
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40
- sudo-bridge-server01 API key: Vault path secret/data/sudo-bridge-server01, field api_key
- sudo-bridge-server01 URL: http://192.168.1.90:8082

## Step-by-step

Write /tmp/hermes_config_update.py and run it. The script should:

1. Get sudo-bridge-server01 API key from Vault (AppRole → read secret → revoke token)

2. Add cat /home/administrator/.hermes/config.yaml to allowlist (tier 1), then execute to read the config

3. Parse the YAML. Find platform_toolsets.telegram. Add hermes-cli if missing.

4. Serialize the updated YAML back to a string.

5. Base64-encode the updated config content. Then use a python3 one-liner to write it:
   - Add to allowlist (tier 2): python3 -c "import base64,builtins; builtins.open('/home/administrator/.hermes/config.yaml','w').write(base64.b64decode('<B64_CONTENT>').decode())"
   - Execute it (user approves in NTFY)

6. Restart Hermes — add to allowlist (tier 1) then execute:
   docker compose -f /opt/appdata/docker/docker-compose/hermes/docker-compose.yml up -d

7. Wait 30s, then verify health:
   docker ps | grep hermes

NOTE: sudo-bridge uses shlex.split() — no shell, no pipes, no &&. Every command is a separate POST /exec call.
NOTE: The python3 -c allowlist pattern should be: python3 -c * (wildcard to match any inline script)

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "confirm hermes-cli now in platform_toolsets.telegram; hermes container healthy"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.
```

---

### AGENT P1-4: Create container-health-sweep SKILL.md

```
You are a documentation agent. Create the container-health-sweep SKILL.md on server-01 so Hermes knows how to sweep Docker containers for health issues.

## Target path on server-01
/home/administrator/.hermes/skills/homelab/container-health-sweep/SKILL.md

## Credentials
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40
- sudo-bridge-server01 API key: Vault path secret/data/sudo-bridge-server01, field api_key
- sudo-bridge-server01 URL: http://192.168.1.90:8082

## SKILL.md content (write exactly this)

---
name: container-health-sweep
description: Sweep all Docker containers and report unhealthy, stopped, or restarting ones
category: homelab
version: "1.0"
trigger:
  - scheduled
  - on_demand
---

# Container Health Sweep

Scans all Docker containers on the host and identifies those that are unhealthy, stopped, or in a restart loop.

## Command

docker ps -a --format "{{.Names}}\t{{.Status}}\t{{.Image}}"

## Health states

- HEALTHY: Up and passing healthcheck
- UNHEALTHY: Running but marked (unhealthy) in status
- STOPPED: Exited state
- RESTARTING: In restart loop (Restarting in status)

## Usage

Hermes calls this skill before triggering a Jenkins redeploy. Output is parsed to identify which containers need attention. Results logged to hermes_redeploy_log table in api_business DB.

## Escalation path

UNHEALTHY detected -> classify failure type -> trigger Jenkins redeploy (secrets-proxy /shell) -> retry up to 3x -> if still failing: docker stop + NTFY user.

Critical services (secrets-proxy, authelia, sudo-bridge, bitwarden-bridge): stop immediately on first failure, no retry.

## Step-by-step

Write /tmp/write_health_skill.py and run it:

1. Get sudo-bridge-server01 API key from Vault (AppRole login → read → revoke)

2. Create directory on server-01:
   - Add mkdir -p /home/administrator/.hermes/skills/homelab/container-health-sweep to allowlist tier 1
   - Execute it

3. Write the SKILL.md using base64 encoding:
   - Encode the SKILL.md content above as base64
   - Add to allowlist (tier 2): python3 -c * (wildcard)
   - Execute: python3 -c "import base64; open('/home/administrator/.hermes/skills/homelab/container-health-sweep/SKILL.md','w').write(base64.b64decode('<B64>').decode())"
   - User approves in NTFY

4. Verify:
   - Add cat /home/administrator/.hermes/skills/homelab/container-health-sweep/SKILL.md to allowlist tier 1
   - Execute and confirm content

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "confirm SKILL.md exists at correct path"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.
```

---

### AGENT P1-5: Create semantic-recall SKILL.md

```
You are a documentation agent. Create the semantic-recall SKILL.md on server-01 so Hermes knows how to search the Obsidian vault semantically.

## Target path on server-01
/home/administrator/.hermes/skills/homelab/semantic-recall/SKILL.md

## Credentials
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40
- sudo-bridge-server01 API key: Vault path secret/data/sudo-bridge-server01, field api_key
- sudo-bridge-server01 URL: http://192.168.1.90:8082

## SKILL.md content (write exactly this)

---
name: semantic-recall
description: Semantic search over the Obsidian vault using nomic-embed-text embeddings
category: homelab
version: "1.0"
trigger:
  - on_demand
---

# Semantic Recall

Searches the Obsidian vault at /opt/appdata/obsidian/vault/ using semantic similarity. Index is a SQLite database maintained by the session-end hook.

## Command

python3 /opt/appdata/docker/.claude/scripts/semantic_recall.py query "<query string>"

## Output format

Returns top matching chunks from Obsidian notes with relevance scores. Each result includes the source file path and the matched text.

## Usage

Call before making infrastructure decisions to retrieve relevant past decisions, session notes, or playbook excerpts.

## Re-index

python3 /opt/appdata/docker/.claude/scripts/semantic_recall.py index

## Step-by-step

Same pattern as container-health-sweep SKILL.md:
1. Get sudo-bridge-server01 API key from Vault
2. mkdir -p /home/administrator/.hermes/skills/homelab/semantic-recall (allowlist tier 1)
3. Write SKILL.md via base64 python3 -c one-liner (allowlist tier 2, user approves in NTFY)
4. Verify with cat (allowlist tier 1)

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "confirm SKILL.md exists at correct path"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.
```

---

## [INLINE after Phase 1 completes]
Trigger jellyfin Jenkins job manually and verify full pipeline succeeds end-to-end:
- Go to jenkins.reverseproxyserver.net → jellyfin job → Build Now
- Watch build log: should complete Build → Push → Sandbox Deploy → Smoke Test → Promote to Production
- If it fails, check the secrets-proxy caller key (id=163 should have fixed this)

---

## PHASE 2 — 5 parallel agents (launch after Phase 1 complete + jellyfin verify)

---

### AGENT P2-1: id=161 — Hermes→Jenkins trigger v1

```
You are a deployment agent implementing the Hermes→Jenkins health-failure trigger (v1). This is a foundational piece of infrastructure — Hermes uses this to auto-redeploy unhealthy containers via Jenkins.

## Task overview
1. Create hermes_redeploy_log table in api_business postgres
2. Generate a hermes caller key, add it to Vault proxy/callers, redeploy secrets-proxy to activate it
3. Write the health-trigger Python script to server-01 at /home/administrator/.hermes/scripts/health_trigger.py
4. Create SKILL.md for the trigger at /home/administrator/.hermes/skills/homelab/health-trigger/SKILL.md

## id=161 v1 design (implement exactly this)
- Trigger: /redeploy <service> Telegram message (manual) OR container UNHEALTHY state (auto)
- Recovery: always full Jenkins redeploy — no restart shortcut
- Critical services (secrets-proxy, authelia, sudo-bridge, bitwarden-bridge): stop immediately + NTFY on first failure, no retry
- Non-critical: 3 Jenkins redeploy attempts, then stop + NTFY if still failing
- Logging: every attempt logged to hermes_redeploy_log table
- Hermes auth: hermes caller key in secrets-proxy (new — must be generated and activated)

## Credentials
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40
- Postgres: docker exec $(docker ps --format '{{.Names}}' | grep '^postgres-') psql -U postgres -d api_business
- secrets-proxy IP: 172.16.16.12:8080 (on coolify network — use docker run --network coolify alpine/curl to reach it)
- sudo-bridge-server01 API key: Vault path secret/data/sudo-bridge-server01, field api_key
- sudo-bridge-server01 URL: http://192.168.1.90:8082

## Step 1: Create hermes_redeploy_log table

Run this SQL via docker exec postgres:

CREATE TABLE IF NOT EXISTS hermes_redeploy_log (
    id SERIAL PRIMARY KEY,
    container_name TEXT NOT NULL,
    failure_type TEXT,
    hermes_diagnosis TEXT,
    attempt_number INTEGER DEFAULT 1,
    triggered_by TEXT NOT NULL,
    outcome TEXT NOT NULL,
    jenkins_build_url TEXT,
    timestamp TIMESTAMPTZ DEFAULT NOW()
);

## Step 2: Generate and store hermes caller key

Write a Python script that:
1. Generates a secure 64-char hex key: import secrets; key = secrets.token_hex(32)
2. AppRole login to Vault
3. Read current secret/data/proxy/callers
4. Add hermes -> key to the dict
5. Write back to Vault: PUT secret/data/proxy/callers with updated dict
6. Revoke token
7. Print "hermes key generated and stored in Vault" (do NOT print the key)

## Step 3: Redeploy secrets-proxy to activate hermes key

Run the same restart_proxy.py pattern from id=163 (already in your instructions above).
The script reads proxy/callers from Vault (now includes hermes) and restarts with updated PROXY_CALLERS.
Script template is identical to P1-1 — write it fresh to /tmp/restart_proxy_161.py and run it.
Verify: curl http://172.16.16.12:8080/health returns 200.

## Step 4: Write health_trigger.py to server-01

Write this script content, base64-encode it, and write to /home/administrator/.hermes/scripts/health_trigger.py via sudo-bridge-server01 tier-2 python3 -c pattern:

SCRIPT CONTENT:
#!/usr/bin/env python3
"""Hermes health trigger v1 — triggers Jenkins redeploy for unhealthy containers."""
import json, sys, subprocess, urllib.request, datetime

CRITICAL_SERVICES = ['secrets-proxy', 'authelia', 'sudo-bridge', 'bitwarden-bridge']
MAX_RETRIES = 3
# Secrets-proxy is on coolify network — reach via docker run
PROXY_URL = 'http://172.16.16.12:8080'

def get_hermes_key():
    """Read hermes caller key from Vault via AppRole."""
    import hashlib
    vault_nets = json.loads(subprocess.check_output(
        "docker inspect vault-iwaulpoi5hwirdlogshmul40 --format '{{json .NetworkSettings.Networks}}'",
        shell=True).decode())
    vault_ip = list(vault_nets.values())[0]['IPAddress']
    role_id = open('/opt/appdata/docker/docker-compose/vault/approle/role-id').read().strip()
    secret_id = open('/opt/appdata/docker/docker-compose/vault/approle/secret-id').read().strip()
    body = json.dumps({'role_id': role_id, 'secret_id': secret_id}).encode()
    req = urllib.request.Request(f'http://{vault_ip}:8200/v1/auth/approle/login',
        data=body, headers={'Content-Type':'application/json'}, method='POST')
    with urllib.request.urlopen(req) as r: tok = json.loads(r.read())['auth']['client_token']
    req = urllib.request.Request(f'http://{vault_ip}:8200/v1/secret/data/proxy/callers',
        headers={'X-Vault-Token': tok})
    with urllib.request.urlopen(req) as r: callers = json.loads(r.read())['data']['data']
    urllib.request.urlopen(urllib.request.Request(
        f'http://{vault_ip}:8200/v1/auth/token/revoke-self', data=b'{}',
        headers={'X-Vault-Token': tok, 'Content-Type':'application/json'}, method='POST'))
    return callers.get('hermes', '')

def proxy_shell(command, key, reason=''):
    body = json.dumps({'command': command, 'target': 'production', 'reason': reason}).encode()
    result = subprocess.run(
        ['docker','run','--rm','--network','coolify','curlimages/curl',
         '-sf','-X','POST','-H','Content-Type: application/json',
         '-H',f'Authorization: Bearer {key}',
         '-d',f'@-', f'{PROXY_URL}/shell'],
        input=body, capture_output=True)
    return json.loads(result.stdout) if result.stdout else {'error': result.stderr.decode()}

def trigger_jenkins(service, key):
    """Trigger Jenkins redeploy via secrets-proxy."""
    cmd = f'curl -sf http://jenkins.reverseproxyserver.net/job/{service}/build'
    return proxy_shell(cmd, key, f'Jenkins redeploy for unhealthy {service}')

def stop_container(container_name):
    subprocess.run(['docker','stop', container_name])

def ntfy_alert(message):
    subprocess.run(['docker','run','--rm','--network','coolify','curlimages/curl',
        '-sf','-X','POST','-d',message,
        'http://ntfy-k0oooo8cckwsok80gg8kck88/hermes-alerts'], capture_output=True)

def main():
    if len(sys.argv) < 2:
        print('Usage: health_trigger.py <container_name> [triggered_by]')
        sys.exit(1)
    container = sys.argv[1]
    triggered_by = sys.argv[2] if len(sys.argv) > 2 else 'auto'

    is_critical = any(svc in container for svc in CRITICAL_SERVICES)
    hermes_key = get_hermes_key()

    if is_critical:
        stop_container(container)
        ntfy_alert(f'CRITICAL: {container} stopped. Interactive session needed immediately.')
        sys.exit(0)

    for attempt in range(1, MAX_RETRIES + 1):
        trigger_jenkins(container, hermes_key)
        import time; time.sleep(60)
        status = subprocess.check_output(
            ['docker','inspect','--format','{{.State.Health.Status}}', container],
            stderr=subprocess.DEVNULL).decode().strip()
        if status == 'healthy':
            ntfy_alert(f'RECOVERED: {container} healthy after {attempt} attempt(s)')
            sys.exit(0)

    stop_container(container)
    ntfy_alert(f'FAILED: {container} still unhealthy after {MAX_RETRIES} redeploys. Stopped. Interactive session needed.')

if __name__ == '__main__':
    main()

## Step 5: Create health-trigger SKILL.md on server-01

Path: /home/administrator/.hermes/skills/homelab/health-trigger/SKILL.md

Content:
---
name: health-trigger
description: Trigger Jenkins redeploy for an unhealthy container; handles retries, critical-service fast-stop, and NTFY alerts
category: homelab
version: "1.0"
---

# Health Trigger

Invoked when Hermes detects a container is unhealthy or receives a /redeploy <service> Telegram command.

## Command

python3 /opt/data/scripts/health_trigger.py <container_name> <triggered_by>

## Examples

python3 /opt/data/scripts/health_trigger.py jellyfin auto
python3 /opt/data/scripts/health_trigger.py n8n telegram_redeploy

## Behavior

- Critical services (secrets-proxy, authelia, sudo-bridge, bitwarden-bridge): immediate stop + NTFY
- Non-critical: up to 3 Jenkins redeploy attempts, then stop + NTFY if still failing
- All attempts logged to hermes_redeploy_log table in api_business DB

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "confirm: table created, hermes key in Vault, secrets-proxy healthy, script at correct path, SKILL.md created"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.
```

---

### AGENT P2-2: id=154 — Domain playbooks → Gitea

```
You are a documentation agent. Push all Claude domain playbooks from the memory directory to a dedicated Gitea repository so agents can reference them via raw URLs.

## Task
1. Check if a Gitea repo named `claude-playbooks` exists under Backtalk6858; create it if not
2. Push all .md files from /home/administrator/.claude/projects/-home-administrator-Desktop-claude/memory/ that are playbooks (filename starts with `playbook_`) to the repo
3. Verify the raw URLs work: https://gitea.reverseproxyserver.net/Backtalk6858/claude-playbooks/raw/branch/main/<filename>

## Credentials
- Gitea admin token: Vault path secret/data/gitea, field admin_token
- Gitea URL: https://gitea.reverseproxyserver.net (or gitea.local for pushes)
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40

## Step-by-step

1. AppRole login → read secret/data/gitea → get admin_token → revoke token

2. Check if repo exists:
   curl -sf -H "Authorization: token <admin_token>" https://gitea.reverseproxyserver.net/api/v1/repos/Backtalk6858/claude-playbooks

3. Create if 404:
   curl -sf -X POST -H "Authorization: token <admin_token>" -H "Content-Type: application/json" \
     -d '{"name":"claude-playbooks","description":"Claude domain playbooks for agent reference","auto_init":true,"default_branch":"main"}' \
     https://gitea.reverseproxyserver.net/api/v1/user/repos

4. Clone locally to /tmp/claude-playbooks, copy all playbook_*.md files from memory dir, commit, push

5. Verify 2-3 random raw URLs return 200

6. Note: memory dir is /home/administrator/.claude/projects/-home-administrator-Desktop-claude/memory/
   Push only files starting with playbook_ (not user_, feedback_, project_, reference_)

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "list of files pushed; repo URL"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.
```

---

### AGENT P2-3: Content repurposing proposal

```
You are a research and writing agent. Research how to repurpose Claude conversation content and session notes into other formats (blog posts, LinkedIn posts, YouTube scripts, etc.). Write a structured proposal and save it as a markdown file.

## Task
1. Research content repurposing workflows — specifically for AI/homelab technical content
2. Assess what raw material exists: Obsidian session notes at /opt/appdata/obsidian/vault/Archives/Sessions/ on server-01, Claude conversation context
3. Propose a content pipeline: what types of content to create, what tools/N8N automations would be needed, estimated effort, platform targets
4. Write the proposal as a well-structured markdown document

## Output
Save the proposal to: /opt/appdata/obsidian/vault/Projects/content-repurposing-proposal.md on server-01

## Context
The user runs a technical homelab and is building AI automation infrastructure. The content has strong LinkedIn/YouTube/blog potential but currently only lives in session notes.

## Credentials
- sudo-bridge-server01 API key: Vault path secret/data/sudo-bridge-server01, field api_key (for writing to server-01)
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40
- sudo-bridge-server01 URL: http://192.168.1.90:8082

## Approach
1. Read a few recent session notes from /opt/appdata/obsidian/vault/Archives/Sessions/ on server-01 to understand the raw material (add cat of a few files to allowlist tier 1, execute)
2. Research content repurposing best practices using WebSearch if available, or reason from your knowledge
3. Write a comprehensive proposal (~800-1200 words) covering:
   - Content types (short-form, long-form, video, written)
   - Platform targets (LinkedIn, YouTube, Blog, Twitter/X)
   - Proposed N8N automation pipeline for extraction + formatting
   - Effort estimates
   - Priority recommendations
4. Write the proposal to server-01 via sudo-bridge tier-2 python3 -c base64 pattern

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "path where proposal was saved; key recommendations"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.
```

---

### AGENT P2-4: id=135 — Update N8N playbook sandbox section

```
You are a documentation agent. Update the N8N playbook's sandbox section to reflect the current state: Vaultwarden is retired, Bitwarden cloud dummy account is used instead, and the current sandbox stack configuration.

## Task
Find and update the sandbox section in the N8N playbook. The playbook is either at:
- /home/administrator/.claude/projects/-home-administrator-Desktop-claude/memory/ (check for playbook_n8n_automations.md or similar)
- /opt/appdata/docker/.claude/playbooks/ on primary

## Current sandbox facts to reflect
- Vaultwarden sandbox REMOVED — Bitwarden cloud dummy account used (megafreeman12@proton.me)
- Sandbox Vault at 192.168.1.90:8201, AppRole at /opt/appdata/docker/docker-compose/server-01/vault-config/
- Sandbox N8N at 192.168.1.90:5679, API key at Vault secret/sandbox/n8n
- Bitwarden bridge sandbox at 192.168.1.90:8083
- postgres-sandbox at default postgres port (no external binding)
- sandbox Bitwarden credentials at Vault secret/sandbox/bitwarden (email, master_password, client_id, client_secret)
- N8N sandbox credentials created: postgres-sandbox, vault-sandbox, n8n-internal-sandbox, Bridge API Key (Sandbox)
- JSON-first rule: always edit workflow JSONs in git repo, import to sandbox N8N for testing; never edit directly in N8N UI
- Sandbox N8N workflow IDs for rotation workflows: o8zxdYbY4Y6JRTEy, 1mM1rC9n2HjHPpJf, lDVXyar70e0w3fs9 (NOT YET TESTED)

## Step-by-step
1. Find the N8N playbook file on primary using the Read or Bash tool
2. Read the current sandbox section
3. Update it to reflect the facts above — do not remove anything that's still accurate
4. Save the file

## Credentials
- No Vault access needed — this is a local file edit on primary

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "playbook path updated; what changed"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.
```

---

### AGENT P2-5: id=112 — Add --model flag to claude -p scripts

```
You are a code update agent. Find all scripts that use `claude -p` and add `--model claude-sonnet-4-6 --thinking medium` flags to each invocation.

## Task
Search the codebase for all files containing `claude -p` invocations. For each one, add the model and thinking flags if not already present.

## Decision locked
Model: claude-sonnet-4-6
Thinking: medium
This is the hardcoded default until id=25 (Cost Intelligence System) is built.

## Search locations
- /opt/appdata/docker/ (all subdirectories)
- /home/administrator/Desktop/claude/ (all subdirectories)
- /opt/appdata/docker/.claude/scripts/
- /home/administrator/.claude/

## Step-by-step
1. Find all files containing `claude -p` using grep -r
2. For each file, read it and identify each `claude -p` invocation
3. Add --model claude-sonnet-4-6 --thinking medium if not already present
4. Preserve all other flags (--max-turns, --output-format, etc.)
5. Save each modified file

## Format
Before: claude -p "prompt text" --max-turns 15
After:  claude -p "prompt text" --model claude-sonnet-4-6 --thinking medium --max-turns 15

Before: claude -p --output-format json
After:  claude -p --model claude-sonnet-4-6 --thinking medium --output-format json

## Important
- Do not modify this prompt file itself (agent_prompts.md)
- Do not modify files in /home/administrator/.claude/projects/.../subagents/ (those are agent output logs)
- If a file already has --model flag, skip it

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["file path — what changed"],
  "actions_failed": ["file — reason"],
  "notes": "total files modified; any files skipped and why"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.
```

---

## PHASE 3 — 1 agent (launch after Phase 2 id=154 completes)

---

### AGENT P3-1: id=155 — required_playbooks precision pass

```
You are a database maintenance agent. The automation_ideas table has a required_playbooks column that was mass-backfilled with Gitea raw URLs by type. Now do a precision pass: verify the URLs actually resolve, and update any rows where the assigned playbooks don't match what the automation actually needs.

## Task
1. Read all rows from automation_ideas where required_playbooks IS NOT NULL
2. For each URL in each row's required_playbooks array, verify it returns 200 (HEAD request to Gitea raw URL)
3. For any URL returning 404, find the correct URL or remove the broken entry
4. For any automation whose task_description clearly references additional playbooks not in required_playbooks, add them
5. Update the DB rows where corrections are made

## Credentials
- Postgres: docker exec $(docker ps --format '{{.Names}}' | grep '^postgres-') psql -U postgres -d api_business
- Gitea token: Vault path secret/data/gitea, field admin_token (for authenticated HEAD requests)
- AppRole: /opt/appdata/docker/docker-compose/vault/approle/role-id + secret-id
- Vault IP: resolve dynamically via docker inspect vault-iwaulpoi5hwirdlogshmul40

## Context
- required_playbooks is a TEXT[] column containing Gitea raw URLs
- The claude-playbooks repo should now exist at: https://gitea.reverseproxyserver.net/Backtalk6858/claude-playbooks/raw/branch/main/
- Backfill was done by automation type (n8n_automation, claude_agent, script) — some may be wrong for specific automations
- Focus on rows with status = 'pending' or 'ready_to_build' first (these are actionable)

## Step-by-step
1. AppRole → Vault → get gitea admin_token → revoke token
2. Query: SELECT id, name, type, task_description, required_playbooks FROM automation_ideas WHERE required_playbooks IS NOT NULL ORDER BY id
3. For each row, HEAD-request each URL with Gitea auth header
4. Log broken URLs, find correct paths in the claude-playbooks repo
5. UPDATE automation_ideas SET required_playbooks = ARRAY[...] WHERE id = ... for each correction
6. Report summary: rows checked, URLs fixed, rows updated

## MANDATORY WRAP-UP

{
  "status": "succeeded|partially_succeeded|failed",
  "actions_taken": ["action — outcome"],
  "actions_failed": ["action — reason"],
  "notes": "rows checked, URLs broken/fixed, rows updated"
}

--max-turns 15
If you issue the same tool call or command twice with identical arguments, STOP immediately and output the mandatory wrap-up with status=partially_succeeded.
```