Chat Prompt Sweep

Runs a corpus of 100 natural-language prompts through the dashboard chat API.

The browser URL is only UI state. The actual command is:

POST http://127.0.0.1:8194/api/chat/ask

Quick Start

Smoke test 5 prompts without executing actions:

python3 examples/44-chat-prompt-sweep/run_chat_prompts.py --limit 5

Run all 100 prompts as dry-run planning/routing tests:

python3 examples/44-chat-prompt-sweep/run_chat_prompts.py

LLM runs send an explicit model in the chat request. The runner resolves it in this order: --model, URIRUN_LLM_MODEL, LLM_MODEL, then the dashboard's non-secret /api/chat/config. Provider quota/rate-limit failures are reported separately as environmentBlocked, without counting as a successful prompt.

Use the current dashboard URL as defaults for host/targets:

python3 examples/44-chat-prompt-sweep/run_chat_prompts.py \
  --from-url 'http://127.0.0.1:8194/?view=chat&tab=chat&execute=1&targets=host&action=chat%3Arun&discovery=node%3Alenovo&prompt=zrob+zrzut+jednego+ekranu++monitora%2C+na+kt%C3%B3rym+jest++przegl%C4%85darka+chrome&prompt_len=37&chat=panel'

Execute accepted flows on the host:

python3 examples/44-chat-prompt-sweep/run_chat_prompts.py --execute --targets host

Cases marked executeAllowed: false are skipped during --execute, unless you also pass --include-side-effects. That protects prompts such as "opublikuj post na LinkedIn" from being executed during a broad sweep.

Useful Slices

Only screen/capture prompts:

python3 examples/44-chat-prompt-sweep/run_chat_prompts.py --category screen --execute

Only routing and node-selection prompts:

python3 examples/44-chat-prompt-sweep/run_chat_prompts.py --category routing

Only Digital Twin prompts:

python3 examples/44-chat-prompt-sweep/run_chat_prompts.py --category twin

Force no-LLM mode:

python3 examples/44-chat-prompt-sweep/run_chat_prompts.py --no-llm

Outputs

Each run creates:

generated/<timestamp>/chat-prompt-results.jsonl — full response per prompt.
generated/<timestamp>/summary.json — aggregate counts by category.
generated/<timestamp>/REPORT.md — human-readable table.
generated/<timestamp>/artifacts/ — materialized inline artifacts, if any.

The console line per prompt includes:

[003/100] ok   screen-003 ok=1 steps=1 target=host 812ms

For deeper diagnosis inspect the JSONL fields:

model, environmentBlocked, generator
selectedTargets, selectedNodes
routingAccepted, routingViolations, runsOnByStep
flowUris, timeline
attachmentCount, artifactCount
humanSignal, degraded, degradedReason

Recommended Gate

For local regression work, use a high but not absolute threshold:

python3 examples/44-chat-prompt-sweep/run_chat_prompts.py \
  --min-ok-rate 0.80

For live host execution, start with a narrow category:

python3 examples/44-chat-prompt-sweep/run_chat_prompts.py \
  --category screen --execute --min-ok-rate 0.70

Files

README.md prompts.json run_chat_prompts.py test_chat_prompt_sweep.py

View on GitHub →