A Functional Taxonomy of World Models

Open original

This essay breaks down the term “world model” by tracing it to the POMDP perception-action loop and identifying its key components—renderers, simulators, planners, and their connecting loop. It shows how different AI fields project parts of this loop to build spatial and temporal understanding beyond language models.

  • The term "world model" is overloaded—video generators with impossible physics and precise physics engines both claim the label, which confuses the field.
  • Decomposing the POMDP loop into renderer (state→observation), simulator (action→state), and planner (observation→action) explains why different fields define "world models" so differently.
  • Computer vision, robotics, and generative AI each project only one slice of this same underlying loop, rather than building genuinely different concepts.

World models in AI shape how machines understand and interact with space and time rather than text. Unlike language models that learn word patterns, world models learn how light, objects and physics behave. Right now everything from video generators that spit out impossible flames to physics engines that nail combustion calls itself a world model. That muddles the field just when precise definitions are critical. At the core lies a loop familiar from reinforcement learning: an agent takes actions, the world changes state, the agent gets observations, and it acts again. State here means a complete snapshot of every object’s position, velocity and properties—even though the agent only ever sees partial data, like pixels or sensor readings. Renderers, simulators and planners each handle one slice of this loop. Renderers turn state into observations, simulators predict how actions change state, and planners pick actions based on past observations. Tracing back to Kenneth Craik’s 1943 idea that minds run “small-scale models” of reality, this framework landed in neural networks in the late 1980s. Modern AI simply reprojects that classic loop in different ways. Recognizing renderers, simulators and planners as distinct parts clears up why computer vision, robotics and generative AI have wildly different definitions of “world model.” Each system taps into the same cycle but focuses on different transitions—state to observation, action to state or observation to action.

Ask about this article