39 — autonomous READ-ONLY browser observation (gillm + LLM vision over URIs)

Capture a node's real screen and let an LLM tell you what's on it — fully autonomous, but observation-only. It uses three local projects as URIs:

• gillm (semcod/gillm) → screen://<node>/portal/query/capture — Wayland screen

capture via the XDG Desktop Portal (the path that works on GNOME/Wayland where gnome-screenshot/x11grab/pipewire return a black frame or hang).

• a vision LLM (LLM_MODEL, e.g. a *image*/*vision* model) → describes the

capture.

• imgl/vql (semcod/imgl, oqlos/vql) can be added as imgl://…/screen/query/layout

for structured OCR (text + element bounding boxes) instead of a prose description.

Verified live on a node ("laptop"): captured 884 KB, and the model reported *"the LinkedIn website, where user Tom Sapletta is logged in … a messaging pane on the right …"* — all read-only.

The read-only gate

observe.py refuses every write/social/login/payment URI before it runs, so an autonomous loop can never turn observation into a post, a message, a like, a login or a purchase — regardless of the goal it's given:

REFUSED = ("/command/", "publish", "form/command/submit", "message", "/send", "dm",
           "comment", "like", "follow", "login", "password", "input/command", "click",
           "type", "pay", "buy", "checkout")

assert_read_only(uri) runs on every call. Reads (screen://…/query/capture, browser://…/page/query/dom) pass; writes (…/form/command/submit, linkedin://post/command/publish) are refused.

Example commands — the autonomous loop

# 0) LLM config from ../.env or ./.env (no manual `set -a; . .env`):
#    LLM_MODEL=openrouter/google/gemini-3.1-flash-image-preview   (a VISION model)
#    OPENROUTER_API_KEY=sk-or-...

# 1) read-only observe: capture the screen + describe it (self-deploys gillm capture if missing)
NODE_URL=http://192.168.188.201:8766 NODE=laptop \
  python3 observe.py

# 2) the LLM "host ask" path, with auto-loaded .env (new --env-file):
urirun host ask laptop "what is on the screen right now" --env-file ../.env --execute

# 3) auto-load ./.env without a flag:
URIRUN_DOTENV=1 urirun host ask laptop "is LinkedIn open and logged in" --execute

# 4) self-managing variant: if the node lacks the capability, it installs it first
NODE_URL=http://192.168.188.201:8766 NODE=laptop \
  python3 ../38-self-managing/run_live.py "capture the screen and tell me what app is open"

Live node smoke test

Current laptop state can be checked without guessing:

curl -fsS --max-time 6 http://192.168.188.201:8766/health
curl -fsS --max-time 6 http://192.168.188.201:8766/routes

On 2026-06-23 the node was up as laptop, running urirun 0.4.56. Before observe.py provisioned screen capture it served five CDP routes:

browser://laptop/cdp/session/command/launch
browser://laptop/cdp/page/command/navigate
browser://laptop/cdp/page/query/eval
browser://laptop/cdp/page/query/screenshot
browser://laptop/cdp/page/query/tabs

After observe.py ran, it merged the portal-capture route too:

screen://laptop/portal/query/capture

That surface is enough to test real browser control over URI:

cd /home/tom/github/if-uri/examples
NODE_URL=http://192.168.188.201:8766 NODE=laptop \
  /home/tom/github/if-uri/urirun/venv/bin/python3 \
  36-remote-browser-cdp/drive_cdp.py https://www.linkedin.com/feed/

Observed result on laptop:

6/6 CDP browser steps ok on laptop
title: LinkedIn Login, Sign in | LinkedIn
links: 24
screenshot: 27879-byte PNG
tabs: LinkedIn Login, Sign in | LinkedIn

This confirms that urirun controls the node browser through browser://.../cdp/*. It also shows that this route launches/uses the CDP profile, not necessarily the physical browser profile where a human is already logged in.

The real monitor capture path was also tested:

cd /home/tom/github/if-uri/examples/39-browser-observe
NODE_URL=http://192.168.188.201:8766 NODE=laptop \
  /home/tom/github/if-uri/urirun/venv/bin/python3 observe.py

Observed result without a configured vision model in the shell:

{
  "ok": true,
  "captured_bytes": 538550,
  "note": "set LLM_MODEL (vision) to analyse the image"
}

Observed result with LLM_MODEL/OPENROUTER_API_KEY loaded from ../.env:

captured_bytes: 538328
observation: terminal application open on a Linux desktop; urirun startup output
             visible, including port 8766 and security warnings.

For full monitor-based autonomy (real desktop screenshot -> OCR/vision -> decision) the node needs one screen/KVM route such as:

screen://laptop/portal/query/capture
browser://laptop/kvm/screen/query/inspect

Without one of those, observe.py self-provisions the screen:// capture handler when /deploy is enabled and the node desktop has python3-dbus and python3-gobject. The CDP smoke test above remains the fallback proof that URI browser control itself works.

What it will and won't do

A "local node" only means the browser runs on your LAN — LinkedIn is still the live public site, so a post/message there is real. This example stays observation-only by design; for a single post you've written, fill the composer and stop before *Publish* so a human clicks it.

Files

• gillm_capture.py — gillm's portal screen capture as a deployable URI handler.
• observe.py — capture → vision-LLM description, with the read-only gate.

Files

View on GitHub →