claude-projects/agent-builder/.claude/playbook_agent_builder_agent.md

Playbook: Agent Builder Agent

Purpose

Builds claude_agent and script type automations from the automation_ideas table. Uses Ollama (llama3.1:8b) as the primary code generator and claude -p as the overseer/validator. Deploys to sandbox, runs all 4 test levels, notifies user for promotion approval.

Trigger

Scheduled or manual. Queries automation_ideas for the next row where:

• type IN ('claude_agent', 'script')
• status = 'ready_to_build'
• builder_status = 'not_started'
• Ordered by priority ASC NULLS LAST, id ASC

Only processes one automation per run.

Infrastructure

• Runs on: server-01 (n8n-sandbox, port 5679)
• Ollama endpoint: http://localhost:11434 (server-01 local)
• Model: llama3.1:8b
• Claude overseer: claude -p (non-interactive, SDK credits — use sparingly)
• Vault: sandbox AppRole at /opt/appdata/docker/docker-compose/vault/approle/
• Database: production api_business (read automation_ideas, write agent_test_results)
• NTFY: production NTFY instance for notifications

Step-by-Step

Step 1 — Claim the automation

UPDATE automation_ideas
SET builder_status = 'queued'
WHERE id = <selected_id> AND builder_status = 'not_started';

If 0 rows updated: another builder claimed it — stop, notify, exit.

Step 2 — Fetch required playbooks

Read the required_playbooks field from the automation row. It contains a comma-separated list of Gitea raw URLs. For each URL:

GET {gitea_raw_url}

Inject the full content of each playbook into the Ollama generation prompt as context. If a URL returns non-200, set builder_status = 'failed', notify via NTFY with the missing URL, stop.

If required_playbooks is empty: proceed without playbook context.

Step 3 — Build the prompt for Ollama

Construct a generation prompt using all available fields from the automation row:

• name: what the automation is called
• task_description: full structured spec — this is the primary instruction
• type: claude_agent or script
• infrastructure_requirement: what infra it needs access to
• Playbook content fetched in Step 2

For claude_agent type, the generated output is a shell-executable claude -p command. The Ollama prompt must include these mandatory requirements for the generated command:

claude_agent MANDATORY PROMPT REQUIREMENTS (all must appear in the generated claude -p system prompt):
1. Hard turn limit: the claude -p call must include --max-turns 15
2. Loop detection instruction: "If you issue the same command twice with identical arguments, STOP immediately and report what you found so far."
3. Credential map: all credentials and access patterns the agent will need must be provided inline in the system prompt — the agent must never search for auth at runtime
4. Defined output: the system prompt must specify the exact output format expected
5. Data pre-fetched: any static data the agent needs (DB rows, config values) must be included inline in the system prompt rather than fetched via tool calls where possible

Ollama prompt structure:

You are an expert automation engineer. Build a {type} automation with the following specification.

Name: {name}
Infrastructure: {infrastructure_requirement}

{playbook_context if any}

Specification:
{task_description}

Requirements:
- If type is claude_agent: output a complete shell-executable claude -p command with full system prompt and all logic. The agent must be self-contained. MUST include --max-turns 15. MUST include loop detection instruction. MUST include credential map inline. MUST include defined output format.
- If type is script: output a complete Python script. Include a Dockerfile if the script has dependencies beyond stdlib.
- Output ONLY the code. No explanation, no markdown fences, no commentary.
- The code must handle its own error cases and log to stdout.
- Secrets must be fetched from Vault via AppRole — never hardcoded. AppRole credentials at /opt/appdata/docker/docker-compose/vault/approle/role-id and secret-id.

Step 4 — Generate with Ollama

POST http://localhost:11434/api/generate
{
  "model": "llama3.1:8b",
  "prompt": "<constructed prompt>",
  "stream": false
}

Set builder_status = 'building' before calling.

If Ollama call fails or times out (>120s): set builder_status = 'failed', log error, notify via NTFY, stop.

Step 5 — Overseer validation with claude -p (for claude_agent type: also validates prompt design rules)

Pass the generated code to claude -p for structural review. Keep the prompt minimal to conserve SDK credits:

claude -p "Review this {type} automation code for the following only:
1. Does it correctly fetch secrets from Vault via AppRole (never hardcoded)?
2. Are there any obvious syntax errors or missing imports?
3. Does the logic match this spec summary: {name} — {task_description[:200]}

Respond with: PASS or FAIL, then one sentence explaining why.
Do not rewrite the code."

If FAIL: log claude's reason, set builder_status = 'failed', notify via NTFY with the failure reason, stop. If PASS: proceed.

Step 6 — Human prompt review gate (claude_agent type only)

For claude_agent type automations, send the full generated claude -p command to the user for review before any testing begins. This is synchronous — do not proceed until the user approves or rejects.

Send NTFY notification:

Title: [Agent Builder] Prompt Review Required — {name}
Body: Overseer passed. Review the generated claude -p prompt before sandbox testing begins.

--- GENERATED COMMAND ---
{generated_claude_p_command}

Reply APPROVE to proceed to testing, or REJECT to cancel.

Wait for user response via NTFY reply or next session confirmation.

• On APPROVE: proceed to Step 7
• On REJECT: set builder_status = 'failed', log rejection, stop. User should update task_description and re-queue.

Skip this step for script type automations.

Step 7 — Deploy to sandbox

For script type:

1. Write the generated code to a temp directory on server-01
2. If a Dockerfile was generated, build the image: docker build -t agent-{id}-{slug} .
3. Run a test container: docker run --rm agent-{id}-{slug} (dry run, no side effects)

For claude_agent type:

1. Write the generated claude -p command to a shell script
2. Make it executable
3. Run it once with --dry-run flag if supported, or with a test input that produces no side effects

If deployment fails: set builder_status = 'failed', log error, notify via NTFY, stop.

Step 8 — Run 4-level automated tests

Run each level in order. Stop and fail if any level fails. Log every result to agent_test_results.

Level 1 — Structure Validate the generated artifact:

• For scripts: python3 -m py_compile script.py — must exit 0
• For claude agents: verify the shell script is syntactically valid bash
• For Dockerfiles: docker build --check if available, else verify FROM and key directives exist
• Insert result: INSERT INTO agent_test_results (automation_id, test_level, status, execution_log) VALUES ({id}, 1, 'pass'/'fail', '{log}')

Level 2 — Deployment

• Verify the artifact can be deployed cleanly (no missing dependencies, image builds successfully, script runs without import errors)
• Must complete without crashing
• Insert result to agent_test_results

Level 3 — Smoke

• Execute the automation with minimal/test inputs
• Must run to completion without an unhandled exception or non-zero exit
• Insert result to agent_test_results

Level 4 — Assertion

• Verify the correct side effect occurred (not string matching — check the actual system state)
• Examples: a file was created, a DB row was written, an API call returned 200, a container is running
• Insert result to agent_test_results

Step 9 — Notify user for promotion approval

If all 4 levels pass:

1. Set builder_status = 'awaiting_approval'
2. Send NTFY notification:

   Title: Agent Ready for Promotion — {name}
   Body: All 4 test levels passed in sandbox. Automation id={id} ({type}) is ready for production promotion. Reply to approve or reject.
   

User must explicitly approve before any production deployment. No auto-promotion in v1.

Step 10 — On approval

Set builder_status = 'approved', then 'deployed' after production deployment completes. Update automation_ideas status = 'deployed'.

Error handling

• Any unhandled exception: set builder_status = 'failed', log to agent_test_results with test_level=0 and status='fail', send NTFY alert
• Always release the claim (reset builder_status to 'not_started') if failing before Step 3 so another run can retry
• After Step 3: leave as 'failed' — requires manual review before retry

NTFY notification patterns

• Build started: [Agent Builder] Building {name} (id={id}, type={type})
• Overseer FAIL: [Agent Builder] FAIL — Overseer rejected {name}: {reason}
• Test level fail: [Agent Builder] FAIL — {name} failed Level {n}: {error}
• Ready for approval: [Agent Builder] READY — {name} passed all tests, awaiting your approval
• Unhandled error: [Agent Builder] ERROR — {name}: {exception}

SDK credit budget

claude -p is called once per automation (Step 4 only). Keep the overseer prompt under 500 tokens. Do not call claude -p for retries or debugging — only for the initial validation pass.