From b99af3d0aa6aeedb2dfeb1e3df75178a8c5482a6 Mon Sep 17 00:00:00 2001 From: Backtalk6858 Date: Fri, 26 Jun 2026 01:31:58 -0500 Subject: [PATCH] feat: add required_playbooks fetch + prompt review gate to both builder playbooks Agent Builder: added Step 2 (fetch required_playbooks from Gitea URLs), added mandatory claude_agent prompt design rules to Ollama prompt (--max-turns 15, loop detection, credential map inline, defined output), added Step 6 synchronous human NTFY gate for claude_agent prompt review before sandbox testing. N8N Builder: same required_playbooks fetch step, added Step 8 synchronous human NTFY gate for any workflow containing a claude -p node. Both changes prevent runaway token loops in built automations by ensuring every generated claude -p call has hard stops baked in and user-reviewed before deploy. Co-Authored-By: Claude Sonnet 4.6 --- .../.claude/playbook_agent_builder_agent.md | 61 ++++++++++++++++--- .../.claude/playbook_n8n_builder_agent.md | 52 +++++++++++++--- 2 files changed, 94 insertions(+), 19 deletions(-) diff --git a/agent-builder/.claude/playbook_agent_builder_agent.md b/agent-builder/.claude/playbook_agent_builder_agent.md index 97bfbc8..4b6a4e9 100644 --- a/agent-builder/.claude/playbook_agent_builder_agent.md +++ b/agent-builder/.claude/playbook_agent_builder_agent.md @@ -31,32 +31,55 @@ WHERE id = AND builder_status = 'not_started'; ``` If 0 rows updated: another builder claimed it — stop, notify, exit. -### Step 2 — Build the prompt for Ollama +### Step 2 — Fetch required playbooks +Read the `required_playbooks` field from the automation row. It contains a comma-separated list of Gitea raw URLs. For each URL: +``` +GET {gitea_raw_url} +``` +Inject the full content of each playbook into the Ollama generation prompt as context. If a URL returns non-200, set builder_status = 'failed', notify via NTFY with the missing URL, stop. + +If `required_playbooks` is empty: proceed without playbook context. + +### Step 3 — Build the prompt for Ollama Construct a generation prompt using all available fields from the automation row: - `name`: what the automation is called - `task_description`: full structured spec — this is the primary instruction - `type`: claude_agent or script - `infrastructure_requirement`: what infra it needs access to +- Playbook content fetched in Step 2 -Prompt structure: +**For `claude_agent` type**, the generated output is a shell-executable `claude -p` command. The Ollama prompt must include these mandatory requirements for the generated command: + +``` +claude_agent MANDATORY PROMPT REQUIREMENTS (all must appear in the generated claude -p system prompt): +1. Hard turn limit: the claude -p call must include --max-turns 15 +2. Loop detection instruction: "If you issue the same command twice with identical arguments, STOP immediately and report what you found so far." +3. Credential map: all credentials and access patterns the agent will need must be provided inline in the system prompt — the agent must never search for auth at runtime +4. Defined output: the system prompt must specify the exact output format expected +5. Data pre-fetched: any static data the agent needs (DB rows, config values) must be included inline in the system prompt rather than fetched via tool calls where possible +``` + +Ollama prompt structure: ``` You are an expert automation engineer. Build a {type} automation with the following specification. Name: {name} Infrastructure: {infrastructure_requirement} +{playbook_context if any} + Specification: {task_description} Requirements: -- If type is claude_agent: output a complete shell-executable claude -p command with full system prompt and all logic. The agent must be self-contained. +- If type is claude_agent: output a complete shell-executable claude -p command with full system prompt and all logic. The agent must be self-contained. MUST include --max-turns 15. MUST include loop detection instruction. MUST include credential map inline. MUST include defined output format. - If type is script: output a complete Python script. Include a Dockerfile if the script has dependencies beyond stdlib. - Output ONLY the code. No explanation, no markdown fences, no commentary. - The code must handle its own error cases and log to stdout. - Secrets must be fetched from Vault via AppRole — never hardcoded. AppRole credentials at /opt/appdata/docker/docker-compose/vault/approle/role-id and secret-id. ``` -### Step 3 — Generate with Ollama +### Step 4 — Generate with Ollama ``` POST http://localhost:11434/api/generate { @@ -69,7 +92,7 @@ Set builder_status = 'building' before calling. If Ollama call fails or times out (>120s): set builder_status = 'failed', log error, notify via NTFY, stop. -### Step 4 — Overseer validation with claude -p +### Step 5 — Overseer validation with claude -p (for claude_agent type: also validates prompt design rules) Pass the generated code to `claude -p` for structural review. Keep the prompt minimal to conserve SDK credits: ```bash @@ -85,7 +108,27 @@ Do not rewrite the code." If FAIL: log claude's reason, set builder_status = 'failed', notify via NTFY with the failure reason, stop. If PASS: proceed. -### Step 5 — Deploy to sandbox +### Step 6 — Human prompt review gate (claude_agent type only) +For `claude_agent` type automations, send the full generated `claude -p` command to the user for review before any testing begins. This is synchronous — do not proceed until the user approves or rejects. + +Send NTFY notification: +``` +Title: [Agent Builder] Prompt Review Required — {name} +Body: Overseer passed. Review the generated claude -p prompt before sandbox testing begins. + +--- GENERATED COMMAND --- +{generated_claude_p_command} + +Reply APPROVE to proceed to testing, or REJECT to cancel. +``` + +Wait for user response via NTFY reply or next session confirmation. +- On APPROVE: proceed to Step 7 +- On REJECT: set builder_status = 'failed', log rejection, stop. User should update task_description and re-queue. + +Skip this step for `script` type automations. + +### Step 7 — Deploy to sandbox **For `script` type:** 1. Write the generated code to a temp directory on server-01 2. If a Dockerfile was generated, build the image: `docker build -t agent-{id}-{slug} .` @@ -98,7 +141,7 @@ If PASS: proceed. If deployment fails: set builder_status = 'failed', log error, notify via NTFY, stop. -### Step 6 — Run 4-level automated tests +### Step 8 — Run 4-level automated tests Run each level in order. Stop and fail if any level fails. Log every result to `agent_test_results`. **Level 1 — Structure** @@ -123,7 +166,7 @@ Validate the generated artifact: - Examples: a file was created, a DB row was written, an API call returned 200, a container is running - Insert result to agent_test_results -### Step 7 — Notify user for promotion approval +### Step 9 — Notify user for promotion approval If all 4 levels pass: 1. Set builder_status = 'awaiting_approval' 2. Send NTFY notification: @@ -134,7 +177,7 @@ If all 4 levels pass: User must explicitly approve before any production deployment. No auto-promotion in v1. -### Step 8 — On approval +### Step 10 — On approval Set builder_status = 'approved', then 'deployed' after production deployment completes. Update automation_ideas status = 'deployed'. diff --git a/agent-builder/.claude/playbook_n8n_builder_agent.md b/agent-builder/.claude/playbook_n8n_builder_agent.md index 1deddec..2ab53ad 100644 --- a/agent-builder/.claude/playbook_n8n_builder_agent.md +++ b/agent-builder/.claude/playbook_n8n_builder_agent.md @@ -48,11 +48,20 @@ WHERE id = AND builder_status = 'not_started'; ``` If 0 rows updated: another builder claimed it — stop, notify, exit. -### Step 2 — Fetch sandbox N8N API key from Vault +### Step 2 — Fetch required playbooks +Read the `required_playbooks` field from the automation row. It contains a comma-separated list of Gitea raw URLs. For each URL: +``` +GET {gitea_raw_url} +``` +Inject the full content of each playbook into the Ollama generation prompt as context in Step 4. If a URL returns non-200, set builder_status = 'failed', notify via NTFY with the missing URL, stop. + +If `required_playbooks` is empty: proceed without playbook context. + +### Step 3 — Fetch sandbox N8N API key from Vault Use sandbox AppRole to read secret/sandbox/n8n. Extract `api_key` and `base_url`. Never log the key value. Pass it in memory only. -### Step 3 — Discover available N8N credentials +### Step 4 — Discover available N8N credentials Before generating, query the sandbox N8N for existing credentials so the generated workflow references them by name: ``` GET {base_url}/api/v1/credentials @@ -60,7 +69,7 @@ X-N8N-API-KEY: {api_key} ``` Extract credential names and types. Pass this list to the Ollama prompt so the generated workflow uses real credential names. -### Step 4 — Build the prompt for Ollama +### Step 5 — Build the prompt for Ollama ``` You are an expert N8N workflow engineer. Generate a valid N8N workflow JSON for the following automation. @@ -81,7 +90,7 @@ Requirements: - The workflow must fully implement the specification — do not stub or placeholder any steps. ``` -### Step 5 — Generate with Ollama +### Step 6 — Generate with Ollama ``` POST http://localhost:11434/api/generate { @@ -97,7 +106,7 @@ Validate it parses as JSON before proceeding. If invalid JSON: set builder_statu If Ollama call fails or times out (>120s): set builder_status = 'failed', log error, notify, stop. -### Step 6 — Overseer validation with claude -p +### Step 7 — Overseer validation with claude -p Pass the generated JSON to `claude -p` for structural review. Keep prompt minimal to conserve SDK credits: ```bash @@ -114,7 +123,30 @@ Do not rewrite the workflow." If FAIL: log claude's reason, set builder_status = 'failed', notify via NTFY with reason, stop. If PASS: proceed. -### Step 7 — Import to sandbox N8N +### Step 8 — Human prompt review gate (claude -p workflows only) +Detect if the workflow calls `claude -p` by scanning the generated JSON for any node containing `claude -p` or `claude_agent` in its parameters. + +If detected: send the full generated workflow JSON to the user for review before importing to sandbox. This is synchronous — do not proceed until the user approves or rejects. + +Send NTFY notification: +``` +Title: [N8N Builder] Prompt Review Required — {name} +Body: Workflow contains claude -p call. Review before sandbox import. + +--- CLAUDE -P NODE PARAMETERS --- +{extracted claude -p node parameters} + +Required checks: --max-turns present, loop detection instruction present, credential map inline, defined output format. + +Reply APPROVE to proceed, or REJECT to cancel. +``` + +- On APPROVE: proceed to Step 9 +- On REJECT: set builder_status = 'failed', log rejection, stop. User should update task_description and re-queue. + +Skip this step for workflows with no claude -p nodes. + +### Step 9 — Import to sandbox N8N ``` POST {base_url}/api/v1/workflows X-N8N-API-KEY: {api_key} @@ -125,14 +157,14 @@ Body: {generated workflow JSON} On success: capture the returned workflow `id` from N8N. Store in notes or a temp variable. On failure (non-2xx): set builder_status = 'failed', log the N8N error response, notify, stop. -### Step 8 — Assign credentials +### Step 10 — Assign credentials For each node in the workflow that references a credential: ``` GET {base_url}/api/v1/workflows/{workflow_id} ``` Verify credential references resolved correctly. If any credential reference is broken (credential name not found), attempt to match by type — if unresolvable, set builder_status = 'failed', notify user with list of missing credentials, stop. -### Step 9 — Run 4-level automated tests +### Step 11 — Run 4-level automated tests Run each level in order. Stop and fail if any level fails. Log every result to `agent_test_results`. **Level 1 — Structure** @@ -162,7 +194,7 @@ Validate the imported workflow via the N8N API: - The specific assertion depends on the automation — derive it from task_description - Insert result to agent_test_results (test_level=4) -### Step 10 — Notify user for promotion approval +### Step 12 — Notify user for promotion approval If all 4 levels pass: 1. Set builder_status = 'awaiting_approval' 2. Send NTFY notification: @@ -173,7 +205,7 @@ If all 4 levels pass: User must explicitly approve before production import. No auto-promotion in v1. -### Step 11 — On approval +### Step 13 — On approval 1. Import the same workflow JSON to production N8N (port 5678) 2. Assign production credentials (different credential names from sandbox) 3. Set builder_status = 'deployed'