More on the topic…
Runway's GWM Worlds 2 generates interactive 3D environments in real time—720p video at 24 fps with synchronized audio at 48 kHz—that respond to your text commands as you explore. You define the initial scene (layout, materials, lighting, characters, physics rules) and then steer it through typed actions like "the character swings a sword" or "water floods the room," plus continuous camera control. The key innovation is that sessions don't follow a fixed script; the world continues from each new input indefinitely, which opens possibilities for games, interactive storytelling, virtual characters, and robot training environments.
The technical foundation is WorldPrompt, a structured format that splits world state into persistent elements (the initial scene description and visual frame) and time-stamped events (actions tied to specific characters or the scene itself). Runway took its existing audio-video generation model and finetuned it on this format, then converted the bidirectional model into an autoregressive one that can generate indefinitely at real-time speeds. The architecture conditions each generation step on three layers: global context (your scene setup), current frame inputs (camera and active text actions), and cached past frames in a sliding window. Video and audio decoders run causally with caching for speed.
The practical reality is messier than the promise. Ahead-of-time authoring—where you write all actions before generation starts—produces higher quality than live real-time control, because detailed prompts can describe entire scenes whereas on-the-fly commands force the model to improvise details. The demo sidesteps this by binding keyboard and mouse inputs to prewritten prompts rather than requiring users to type. The system does work for agentic control; Runway tested it with a simulated robot that successfully navigated to colored flags based on instructions, suggesting the environment is usable for agent training and evaluation.
Questions about this article
No questions yet.