52 — office work on a virtual machine over RDP, surfaced through noVNC

A user asks for an office task on a remote VM in plain language; a planner turns it into a multi-step URI flow over a VM-office MCP tool surface; urirun executes the flow step by step against a fleet of virtual machines reached over RDP and viewed through an HTML5 noVNC canvas; then a verification step checks the system state — not only that the task happened, but that the RDP session and noVNC view were torn down cleanly (no dangling connections).

This is the everyday corporate pattern: a worker (or an agent) connects to a Windows/Linux VM over RDP, the session is surfaced through noVNC in the browser, and office apps (Excel / Outlook / a billing form) run *inside the VM*.

NL request ──► action space = 20 MCP tools (URIs + JSON Schemas)
          ──► plan: [{uri, payload}, …]   (≥10 steps; deterministic, or --llm)
          ──► urirun.run each step (policy-gated)
                vm://    fleet power (start/stop/list)
                rdp://   gateway session (connect/disconnect/list)
                novnc:// HTML5 view over the session (open/close/status)
                desktop://, screen://, fs://, clipboard://, notify://  — inside the VM
          ──► verify(state): did the task happen AND did RDP/noVNC close cleanly?

Six tasks, each a ≥10-step flow that drives a VM end to end:

Run

python3 run.py                 # all 6 scenarios, deterministic plans, with verification
python3 run.py --scenario finance
python3 run.py --json          # machine-readable

VM-office MCP tool surface: 20 URI tools (schemes: clipboard, desktop, fs, notify, novnc, rdp, screen, vm)
✓ [finance] 13 steps — Connect to the Win11 finance VM over RDP/noVNC, enter Q3 revenue in Excel, save & verify
    executed 13/13 ok=True  ·  verified: True (ocr=True saved=True sessions=0 views=0)
...
RESULT: 6/6 VM-office tasks completed AND verified

Drive a REAL noVNC desktop

--live runs the headline finance task against an actual noVNC desktop in Docker, reusing example 28's connector — proving the same plan shape (start → launch → type → screenshot → stop) drives a real HTML5 canvas, not just the simulator:

python3 run.py --live          # needs Docker; pulls dorowu/ubuntu-desktop-lxde-vnc

Plan with a real LLM

--llm sends the NL request plus the MCP tool schemas to a model (LLM_MODEL + key from examples/.env) and executes the plan it returns, then verifies:

python3 run.py --scenario finance --llm

The verification is the point: a model that connects before powering the VM on, or that forgets to disconnect, is reported verified: False (dangling session), not silently "done". The deterministic flows in scenarios.py are the reference of what a capable model produces.

Why this examines *correct behaviour*, not just "it ran"

The verification on each scenario asserts teardown (sessions == 0, views == 0), and test_vm_office.py exercises the state machine directly:

• connecting to a powered-off VM is rejected (start it before connecting);
• opening a noVNC view requires an RDP session first;
• disconnect closes the dependent noVNC views; power-off tears down the VM's

sessions — no dangling artefacts either way;

• the RDP clipboard is shared across gateway sessions (redirection);
• files persist on the VM across reconnects.

These are the things that go wrong on real RDP/noVNC gateways, so they are what the example checks.

How it maps to a real machine

vm_office_system.py keeps a JSON state (VM power, RDP sessions, noVNC views, the shared clipboard, per-VM files, an OCR log). Every route mutates it, so step N sees step N-1 and verify() checks the real outcome. Swap these routes for the tellmesh desktop connectors (rdp/kvm/novnc/uriscreen — see example 31) or the live noVNC connector from example 28 and the same flows drive an actual VM. Because they are ordinary URI routes, they also run on a remote, NAT'd gateway through mesh.urirun.com exactly like example 32.

Files

• vm_office_system.py — the VM-fleet + RDP/noVNC simulator and the 20-route MCP tool surface.
• scenarios.py — the 6 NL tasks: request, the ≥10-step flow, and verify(state) (incl. teardown).
• run.py — planner (deterministic, --llm, or --live on a real desktop) → execute → verify → report.
• test_vm_office.py — every task plans ≥10 steps, executes and verifies; plus direct behaviour checks; opt-in live (URIRUN_NOVNC_LIVE=1).

Test

python3 -m pytest test_vm_office.py -q

Files

View on GitHub →