My voice harness is my new operating system

Open original

The author built a voice-driven web runtime where a local daemon bridges a live Gemini model to the browser and the OS, letting them run sandboxed mini-apps, edit project files, and hand tasks to other agent harnesses. It's a working prototype of what an AI-first operating system might look like, with the prompt-injection risks left unsolved.

  • Voice is faster than typing for loading context into an LLM, so the author built a voice harness that listens while they keep working and can be interrupted mid-response.
  • Architecture: a browser page captures mic and screen, a local node daemon holds the API key and runs tools, and file writes route to OPFS, a picked folder, or disk depending on where the project lives.
  • Mini-apps are model-written HTML in two nested iframes with a locked-down sandbox, and extensions are JSON descriptors over six primitives so the model never sees API keys.
  • The author has no safeguards yet against prompt injection; anything on a shared screen or web page can reach the model and trigger tool calls to email and system settings.

The author built a voice-driven "harness" that lets them control their computer entirely by speaking, and that runs a small browser-based operating layer on top of it. The setup has three parts: a browser page (which the author calls a "room") as the interface, a local Node daemon (server.mjs) that runs the agent loop and executes tools, and a cloud-hosted live model, which the author names as Gemini-3.8-live, streamed over WebSockets. Mic audio goes through an AudioWorklet to the daemon, which holds the API key and relays every tool call the model makes. The author's motivation is that they think faster than they can type, and they want something that can listen while they keep working and pick up tasks asynchronously. The most interesting engineering is in how files and apps are handled. A project can live in browser-private storage (OPFS), in a folder the user picked through the File System Access API, or on disk at a path the daemon can reach. The model doesn't care which; a router in the daemon decides where a write should go. Browser-side folders can't be touched by the daemon directly (a folder handle exposes no path), so writes get relayed back to the page, which re-checks that the path stays inside the project and logs the change. Calls that nobody can reach are refused with a reason rather than failing silently. Projects can also carry an AGENTS.md whose nearest copy gets injected into the system prompt. On top of this, the model can write complete HTML "mini-apps" that get saved to a shelf and reopened later. These run in two nested iframes: an outer same-origin bridge that vets messages, and an inner sandboxed frame with an opaque origin, no storage, and a CSP that blocks the network by default. Apps expose tools through WebMCP (document.modelContext.registerTool), so the model can call into them like any other tool. The author also adds screen sharing via getDisplayMedia, so the model can see what's on screen, and a Picture-in-Picture window to show when the harness is acting. Extensions are declarative JSON descriptors that parameterize six primitives (http-get, read-file, write-file, list-files, now, wasm), so a Brave Search integration is a few lines of config with a rate limit and an injected API key. The author is candid about rough edges: overlapping windows, a screen-capture bug, and LLM-generated UI that still needs cleaning up. The useful takeaway is less the voice demo and more the security model. The daemon-plus-sandboxed-iframe design is a reasonable template for letting model-written code run locally without handing it the machine, and the author's list of what the web platform still lacks is worth reading on its own.

Ask about this article