feat: add required_playbooks fetch + prompt review gate to both builder playbooks

Agent Builder: added Step 2 (fetch required_playbooks from Gitea URLs), added
mandatory claude_agent prompt design rules to Ollama prompt (--max-turns 15,
loop detection, credential map inline, defined output), added Step 6 synchronous
human NTFY gate for claude_agent prompt review before sandbox testing.

N8N Builder: same required_playbooks fetch step, added Step 8 synchronous human
NTFY gate for any workflow containing a claude -p node.

Both changes prevent runaway token loops in built automations by ensuring every
generated claude -p call has hard stops baked in and user-reviewed before deploy.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Backtalk6858
2026-06-26 01:31:58 -05:00
parent 924e7d5e8b
commit b99af3d0aa
2 changed files with 94 additions and 19 deletions
@@ -31,32 +31,55 @@ WHERE id = <selected_id> AND builder_status = 'not_started';
```
If 0 rows updated: another builder claimed it — stop, notify, exit.
### Step 2 — Build the prompt for Ollama
### Step 2 — Fetch required playbooks
Read the `required_playbooks` field from the automation row. It contains a comma-separated list of Gitea raw URLs. For each URL:
```
GET {gitea_raw_url}
```
Inject the full content of each playbook into the Ollama generation prompt as context. If a URL returns non-200, set builder_status = 'failed', notify via NTFY with the missing URL, stop.
If `required_playbooks` is empty: proceed without playbook context.
### Step 3 — Build the prompt for Ollama
Construct a generation prompt using all available fields from the automation row:
- `name`: what the automation is called
- `task_description`: full structured spec — this is the primary instruction
- `type`: claude_agent or script
- `infrastructure_requirement`: what infra it needs access to
- Playbook content fetched in Step 2
Prompt structure:
**For `claude_agent` type**, the generated output is a shell-executable `claude -p` command. The Ollama prompt must include these mandatory requirements for the generated command:
```
claude_agent MANDATORY PROMPT REQUIREMENTS (all must appear in the generated claude -p system prompt):
1. Hard turn limit: the claude -p call must include --max-turns 15
2. Loop detection instruction: "If you issue the same command twice with identical arguments, STOP immediately and report what you found so far."
3. Credential map: all credentials and access patterns the agent will need must be provided inline in the system prompt — the agent must never search for auth at runtime
4. Defined output: the system prompt must specify the exact output format expected
5. Data pre-fetched: any static data the agent needs (DB rows, config values) must be included inline in the system prompt rather than fetched via tool calls where possible
```
Ollama prompt structure:
```
You are an expert automation engineer. Build a {type} automation with the following specification.
Name: {name}
Infrastructure: {infrastructure_requirement}
{playbook_context if any}
Specification:
{task_description}
Requirements:
- If type is claude_agent: output a complete shell-executable claude -p command with full system prompt and all logic. The agent must be self-contained.
- If type is claude_agent: output a complete shell-executable claude -p command with full system prompt and all logic. The agent must be self-contained. MUST include --max-turns 15. MUST include loop detection instruction. MUST include credential map inline. MUST include defined output format.
- If type is script: output a complete Python script. Include a Dockerfile if the script has dependencies beyond stdlib.
- Output ONLY the code. No explanation, no markdown fences, no commentary.
- The code must handle its own error cases and log to stdout.
- Secrets must be fetched from Vault via AppRole — never hardcoded. AppRole credentials at /opt/appdata/docker/docker-compose/vault/approle/role-id and secret-id.
```
### Step 3 — Generate with Ollama
### Step 4 — Generate with Ollama
```
POST http://localhost:11434/api/generate
{
@@ -69,7 +92,7 @@ Set builder_status = 'building' before calling.
If Ollama call fails or times out (>120s): set builder_status = 'failed', log error, notify via NTFY, stop.
### Step 4 — Overseer validation with claude -p
### Step 5 — Overseer validation with claude -p (for claude_agent type: also validates prompt design rules)
Pass the generated code to `claude -p` for structural review. Keep the prompt minimal to conserve SDK credits:
```bash
@@ -85,7 +108,27 @@ Do not rewrite the code."
If FAIL: log claude's reason, set builder_status = 'failed', notify via NTFY with the failure reason, stop.
If PASS: proceed.
### Step 5Deploy to sandbox
### Step 6Human prompt review gate (claude_agent type only)
For `claude_agent` type automations, send the full generated `claude -p` command to the user for review before any testing begins. This is synchronous — do not proceed until the user approves or rejects.
Send NTFY notification:
```
Title: [Agent Builder] Prompt Review Required — {name}
Body: Overseer passed. Review the generated claude -p prompt before sandbox testing begins.
--- GENERATED COMMAND ---
{generated_claude_p_command}
Reply APPROVE to proceed to testing, or REJECT to cancel.
```
Wait for user response via NTFY reply or next session confirmation.
- On APPROVE: proceed to Step 7
- On REJECT: set builder_status = 'failed', log rejection, stop. User should update task_description and re-queue.
Skip this step for `script` type automations.
### Step 7 — Deploy to sandbox
**For `script` type:**
1. Write the generated code to a temp directory on server-01
2. If a Dockerfile was generated, build the image: `docker build -t agent-{id}-{slug} .`
@@ -98,7 +141,7 @@ If PASS: proceed.
If deployment fails: set builder_status = 'failed', log error, notify via NTFY, stop.
### Step 6 — Run 4-level automated tests
### Step 8 — Run 4-level automated tests
Run each level in order. Stop and fail if any level fails. Log every result to `agent_test_results`.
**Level 1 — Structure**
@@ -123,7 +166,7 @@ Validate the generated artifact:
- Examples: a file was created, a DB row was written, an API call returned 200, a container is running
- Insert result to agent_test_results
### Step 7 — Notify user for promotion approval
### Step 9 — Notify user for promotion approval
If all 4 levels pass:
1. Set builder_status = 'awaiting_approval'
2. Send NTFY notification:
@@ -134,7 +177,7 @@ If all 4 levels pass:
User must explicitly approve before any production deployment. No auto-promotion in v1.
### Step 8 — On approval
### Step 10 — On approval
Set builder_status = 'approved', then 'deployed' after production deployment completes.
Update automation_ideas status = 'deployed'.