47 — NL → ui:// plan → execute (autonomous desktop control)
Turn a natural-language desktop task into a sequence of kvm://.../ui/* (+ browser/app) URIs and run them on a urirun node with a perceive→act→verify loop — the planner that closes the autonomy loop on top of the cross-platform urirun-connector-kvm semantic UI layer.
# plan only (prints the URI steps the LLM produced)
python run.py "focus the terminal and type: echo hello"
# plan + execute on a node, saving a host-side screenshot (+ zoom around each click) per step
python run.py "open a new terminal and run echo hi" \
--node http://192.168.188.201:8765 --execute \
--identity ~/.ssh/id_ed25519 \
--save-shots ~/.urirun/artifacts/kvm-laptopLLM via examples/.env (LLM_MODEL / OPENROUTER_API_KEY), in-process (litellm). The action space is the connector's ui/query/find|wait|verify, ui/command/click|fill, input/*, app://.../launch, browser://.../open.
--save-shots DIR writes HHMMSS-stepNN-<route>.png after every step (and a ...-zoom.png tile around the click point when the step has x,y) so a session is reviewable on the host immediately.
Grounding caveats (read before trusting clicks)
The plan/executor is reliable; the *grounding* depends on the target environment:
- Element targeting is rock-solid only when the app exposes accessibility (AT-SPI on
Linux). For Chrome/Firefox web content, enable it (chrome://accessibility → Native API on, or --force-renderer-accessibility) — otherwise the connector falls back to vision (imgl/vql), whose OCR is unreliable on dark-theme UIs.
- **Coordinate clicks (
ui/command/clickwith x,y, vision fallback) are unreliable on
multi-monitor + HiDPI Wayland: ydotool absolute positioning does not map cleanly to screenshot pixels there (calibration on a 3200×3800 / 2× HiDPI / dual-monitor host showed neither ×1 nor ÷2 mapping landed). Prefer AT-SPI element actions, a single-monitor / non-HiDPI target, or a vdisplay** owned virtual display for deterministic coordinates.
- GNOME Wayland forbids focus-stealing: keyboard goes to the *active* window only.
Launch a fresh window (active on launch) or activate the target before typing.