claude-projects/agent-builder/.claude/playbook_agent_builder_agent.md

Playbook: Agent Builder Agent

Purpose

Builds claude_agent and script type automations from the automation_ideas table. Uses Ollama (llama3.1:8b) as the primary code generator and claude -p as the overseer/validator. Deploys to sandbox, runs all 4 test levels, notifies user for promotion approval.

Trigger

Scheduled or manual. Queries automation_ideas for the next row where:

• type IN ('claude_agent', 'script')
• status = 'ready_to_build'
• builder_status = 'not_started'
• Ordered by priority ASC NULLS LAST, id ASC

Only processes one automation per run.

Infrastructure

• Runs on: server-01 (n8n-sandbox, port 5679)
• Ollama endpoint: http://localhost:11434 (server-01 local)
• Model: llama3.1:8b
• Claude overseer: claude -p (non-interactive, SDK credits — use sparingly)
• Vault: sandbox AppRole at /opt/appdata/docker/docker-compose/vault/approle/
• Database: production api_business (read automation_ideas, write agent_test_results)
• NTFY: production NTFY instance for notifications

Step-by-Step

Step 1 — Claim the automation

UPDATE automation_ideas
SET builder_status = 'queued'
WHERE id = <selected_id> AND builder_status = 'not_started';

If 0 rows updated: another builder claimed it — stop, notify, exit.

Step 2 — Fetch required playbooks

Read the required_playbooks field from the automation row. It contains a comma-separated list of Gitea raw URLs. For each URL:

GET {gitea_raw_url}

Inject the full content of each playbook into the Ollama generation prompt as context. If a URL returns non-200, set builder_status = 'failed', notify via NTFY with the missing URL, stop.

If required_playbooks is empty: proceed without playbook context.

Step 3 — Build the prompt for Ollama

Construct a generation prompt using all available fields from the automation row:

• name: what the automation is called
• task_description: full structured spec — this is the primary instruction
• type: claude_agent or script
• infrastructure_requirement: what infra it needs access to
• Playbook content fetched in Step 2

For claude_agent type, the generated output is a shell-executable claude -p command. The Ollama prompt must include these mandatory requirements for the generated command:

claude_agent MANDATORY PROMPT REQUIREMENTS (all must appear in the generated claude -p system prompt):
1. Hard turn limit: the claude -p call must include --max-turns 15
2. Loop detection instruction: "If you issue the same command twice with identical arguments, STOP immediately and report what you found so far."
3. Credential map: all credentials and access patterns the agent will need must be provided inline in the system prompt — the agent must never search for auth at runtime
4. Defined output: the system prompt must specify the exact output format expected
5. Data pre-fetched: any static data the agent needs (DB rows, config values) must be included inline in the system prompt rather than fetched via tool calls where possible

Ollama prompt structure:

You are an expert automation engineer. Build a {type} automation with the following specification.

Name: {name}
Infrastructure: {infrastructure_requirement}

{playbook_context if any}

Specification:
{task_description}

Requirements:
- If type is claude_agent: output a complete shell-executable claude -p command with full system prompt and all logic. The agent must be self-contained. MUST include --max-turns 15. MUST include loop detection instruction. MUST include credential map inline. MUST include defined output format.
- If type is script: output a complete Python script. Include a Dockerfile if the script has dependencies beyond stdlib.
- Output ONLY the code. No explanation, no markdown fences, no commentary.
- The code must handle its own error cases and log to stdout.
- Secrets must be fetched from Vault via AppRole — never hardcoded. AppRole credentials at /opt/appdata/docker/docker-compose/vault/approle/role-id and secret-id.

Step 4 — Generate with Ollama

POST http://localhost:11434/api/generate
{
  "model": "llama3.1:8b",
  "prompt": "<constructed prompt>",
  "stream": false
}

Set builder_status = 'building' before calling.

If Ollama call fails or times out (>120s): set builder_status = 'failed', log error, notify via NTFY, stop.

Step 5 — Overseer validation with claude -p (for claude_agent type: also validates prompt design rules)

Pass the generated code to claude -p for structural review. Keep the prompt minimal to conserve SDK credits:

claude -p --model claude-sonnet-4-6 --thinking medium "Review this {type} automation code for the following only:
1. Does it correctly fetch secrets from Vault via AppRole (never hardcoded)?
2. Are there any obvious syntax errors or missing imports?
3. Does the logic match this spec summary: {name} — {task_description[:200]}

Respond with: PASS or FAIL, then one sentence explaining why.
Do not rewrite the code."

If FAIL: log claude's reason, set builder_status = 'failed', notify via NTFY with the failure reason, stop. If PASS: proceed.

Step 6 — Human prompt review gate (claude_agent type only)

For claude_agent type automations, send the full generated claude -p command to the user for review before any testing begins. This is synchronous — do not proceed until the user approves or rejects.

Send NTFY notification:

Title: [Agent Builder] Prompt Review Required — {name}
Body: Overseer passed. Review the generated claude -p prompt before sandbox testing begins.

--- GENERATED COMMAND ---
{generated_claude_p_command}

Reply APPROVE to proceed to testing, or REJECT to cancel.

Wait for user response via NTFY reply or next session confirmation.

• On APPROVE: proceed to Step 7
• On REJECT: set builder_status = 'failed', log rejection, stop. User should update task_description and re-queue.

Skip this step for script type automations.

Step 7 — Deploy to sandbox

For script type:

1. Write the generated code to a temp directory on server-01
2. If a Dockerfile was generated, build the image: docker build -t agent-{id}-{slug} .
3. Run a test container: docker run --rm agent-{id}-{slug} (dry run, no side effects)

For claude_agent type:

1. Write the generated claude -p command to a shell script
2. Make it executable
3. Run it once with --dry-run flag if supported, or with a test input that produces no side effects

If deployment fails: set builder_status = 'failed', log error, notify via NTFY, stop.

Step 8 — Run 4-level automated tests

Run each level in order. Stop and fail if any level fails. Log every result to agent_test_results.

Level 1 — Structure Validate the generated artifact:

• For scripts: python3 -m py_compile script.py — must exit 0
• For claude agents: verify the shell script is syntactically valid bash
• For Dockerfiles: docker build --check if available, else verify FROM and key directives exist
• Insert result: INSERT INTO agent_test_results (automation_id, test_level, status, execution_log) VALUES ({id}, 1, 'pass'/'fail', '{log}')

Level 2 — Deployment

• Verify the artifact can be deployed cleanly (no missing dependencies, image builds successfully, script runs without import errors)
• Must complete without crashing
• Insert result to agent_test_results

Level 3 — Smoke

• Execute the automation with minimal/test inputs
• Must run to completion without an unhandled exception or non-zero exit
• Insert result to agent_test_results

Level 4 — Assertion

• Verify the correct side effect occurred (not string matching — check the actual system state)
• Examples: a file was created, a DB row was written, an API call returned 200, a container is running
• Insert result to agent_test_results

Step 9 — Notify user for promotion approval

If all 4 levels pass:

1. Set builder_status = 'awaiting_approval'
2. Send NTFY notification:

   Title: Agent Ready for Promotion — {name}
   Body: All 4 test levels passed in sandbox. Automation id={id} ({type}) is ready for production promotion. Reply to approve or reject.
   

User must explicitly approve before any production deployment. No auto-promotion in v1.

Step 10 — On approval

Set builder_status = 'approved', then 'deployed' after production deployment completes. Update automation_ideas status = 'deployed'.

Design Principle: Structural Wrap-up

Wrap-up steps must be structural constraints, not prompt instructions. An agent instructed to "remember to wrap up" will skip those steps under context pressure, error conditions, or LLM divergence. Structural enforcement means the workflow cannot physically reach completion without passing through each wrap-up step.

Required wrap-up sequence (every automation must include these):

1. DB status update — set builder_status to final state (deployed / failed / awaiting_approval)
2. NTFY notification — send final status to user
3. Error log — write execution summary to agent_test_results (even on success)

For N8N workflows: These must be three separate nodes wired in sequence before the End node. The workflow graph enforces them — they cannot be skipped.

For claude_agent type: The system prompt must list these as required FINAL TOOL CALLS in the output format definition — not as "remember to" instructions. Example: "Your final response MUST include: (1) a DB status update, (2) an NTFY notification, (3) a final status JSON. Do not stop before completing all three."

For script type: Wrap-up must be in a finally: block — never in the main try: body where an exception can skip it.

The test: If you can imagine the automation completing its main task and then stopping before wrap-up, the wrap-up is not structural. Fix it before shipping.

Error handling

• Any unhandled exception: set builder_status = 'failed', log to agent_test_results with test_level=0 and status='fail', send NTFY alert
• Always release the claim (reset builder_status to 'not_started') if failing before Step 3 so another run can retry
• After Step 3: leave as 'failed' — requires manual review before retry

NTFY notification patterns

• Build started: [Agent Builder] Building {name} (id={id}, type={type})
• Overseer FAIL: [Agent Builder] FAIL — Overseer rejected {name}: {reason}
• Test level fail: [Agent Builder] FAIL — {name} failed Level {n}: {error}
• Ready for approval: [Agent Builder] READY — {name} passed all tests, awaiting your approval
• Unhandled error: [Agent Builder] ERROR — {name}: {exception}

SDK credit budget

claude -p is called once per automation (Step 4 only). Keep the overseer prompt under 500 tokens. Do not call claude -p for retries or debugging — only for the initial validation pass.