WorldTUI: I trained a 360M model to pretend to be a terminal
The author fine-tuned a small 360M-parameter language model to simulate a terminal with no real shell behind it, predicting how the screen changes after each keypress. It can create files, run vim and tmux, and play snake, though it only sometimes remembers earlier commands.
- Instead of generating text token by token, the model predicts screen updates, with a COPY operation for scrolling that cuts a full redraw from roughly 3,000 tokens to about three tokens per row.
- Training on its own mistakes (a DAgger-style repair approach) raised PresetBench from 24/47 to 38/47 at int4.
- Decoding rows in parallel brought full-screen redraws from up to 12 seconds down to something usable, making near-realtime performance possible on consumer hardware.
- Memory is still unreliable: the model read back correct values in only 9 of 13 file tests, and combining commands in new ways often breaks.
The author fine-tuned Qwen2.5-Coder-0.5B, a 360M-parameter model, to behave like a terminal with no shell or filesystem behind it. Every keypress goes in as raw bytes, and the model predicts which screen rows change and where the cursor lands. It can create files, run vim and tmux, play snake, and mostly remember earlier input. The project is a scaled-down test of Genie 3's idea of learned applications: instead of a game engine rendering frames, a model predicts the next screen state from user input. A terminal suits this because it's a grid of characters, and training data can be generated by running real shells in Docker and recording every keystroke alongside the resulting screen.
The representation is the interesting engineering. Rather than predicting a text stream, the model outputs a compact list of row rewrites and a cursor position, drawing over old characters without clearing first. Qwen's existing byte tokens are reused for characters and coordinates, giving a vocabulary of just 333 tokens, with grammar-constrained decoding so every output is a valid screen update. Scrolling would cost roughly 96 tokens per row if redrawn in full, so the author added a COPY operation that brings it down to three tokens per row. Training data came to 336k synthetic episodes, about 2B tokens, with randomized usernames, terminal sizes, and file layouts, and 35% of sessions starting mid-application. Compare that to the base model's roughly 18T tokens of pretraining.
The most useful technical lesson is about error compounding. Plain supervised fine-tuning let a single misspelled filename propagate through a session. The fix borrowed from DAgger: let the model run a few steps from real states, compare its screen to the recorded one, and train it to repair its own mistakes. A constraint keeps this honest: corrections may only use words that appeared on real screens or in user input, so the model isn't rewarded for hallucinating. About a third of sampled sessions yielded usable corrections, run over three rounds. On the author's PresetBench, this raised int4 accuracy from 24/47 to 38/47. The whole demo is a 208 MB download that runs in-browser via WebGPU, best in desktop Chrome or Edge, and the author's own test is to type "pythoon" and see that nothing real is running behind it.