Hold That Thought: A Somatic-Symbolic-Semantic Architecture for Shaping Ideas in Real Time

Open original

This is Chuanqi Sun's MIT Media Lab master's thesis proposal arguing that real-time AI should be used through physical handles (tangible pucks, gestures, sliders) bound to meaning, rather than through turn-based chat. It describes the S3 architecture and a set of systems built or planned to test whether this keeps people more creative, in control, and engaged.

  • Chat-style AI is indirect, pliant, and monistic, which the author argues erodes agency, creativity, and joy; the thesis proposes closing the loop in real time so people can hold and move ideas directly.
  • S3 defines three nodes (somatic body, symbolic code-interpreted parameters, semantic model-interpreted meaning) and sets a real-time bar of at least 24 fps and no more than 100 ms input-to-response.
  • The work includes completed systems (Dream Brush, Parametric Diffusion, Spaceship, where Spaceship's demo drew about one million Instagram views) and planned ones (Puck Grammar, a tabletop IDE, an actuated puck table, Sable), evaluated against chat using the Creativity Support Index.

Chuanqi Sun's MIT Media Lab thesis proposal argues that today's AI is stuck in turn-taking: you prompt, wait, judge, repeat. That makes AI indirect (intent has to squeeze through a prompt), pliant (models tend to agree with whatever you assert, so there's nothing to push against), and monistic (outputs converge on one answer, even as they make individuals more creative). The proposed fix is S3, a three-node architecture linking the somatic (the body), the symbolic (things code interprets, like parameters, geometry, and puck positions), and the semantic (things a model interprets, including typed prompts). Direct Manipulation already connects body and symbol, and Interface Agents connect body and meaning but skip symbols. S3 routes the body through symbols bound to meaning, so a hand can hold an idea as a handle. Sun's central hypothesis is that when people work through the symbolic node and every loop closes in real time, they keep more agency, create more, and feel more joy than when chatting with AI in turns. He defines real time as at least 24 frames per second of visible output and no more than 100 ms from input to response, and treats this as a floor, since direct touch demands more. The proposal rests on two finished systems: Dream Brush, a calligraphic canvas using diffusion where a stroke carries both meaning and place, and Parametric Diffusion, plus a screen IDE called Spaceship. The planned work is more ambitious: Puck Grammar, a tangible language where people arrange pucks to express verbs like steer, blend, bind, and lock; Put That There Again, a tabletop IDE fed by a input method called Braindump; an actuated puck table where the AI can move and resist handles; and Sable, a tangible audio synthesizer. Sun is candid that current models are often too slow for these loops, so he plans Wizard-of-Oz and simulated-latency layers and treats latency as an experimental variable. Evaluation is the part that will decide whether any of this matters. Sun plans to compare S3 systems against chat using the Creativity Support Index and measures of agency, ownership, and joy. The thesis is worth reading for its framing more than its current results, since it is a proposal and the core claims are untested. Its most useful idea is the distinction between a prompt, which asks a model for an answer, and a handle, which lets a person keep shaping an idea while the machine responds.

Ask about this article