What objective is reasoning solving?

Open original

The article reframes LLM reasoning as searching through token space for paths that land on a correct answer, using a verifier to define success. It argues that RL training is essentially rejection-sampling fine-tuning on the model's own successful paths, which explains why chains of thought grow longer and why individual paths can still go wrong.

  • The success probability is the sum of probabilities over all paths that reach an accepted answer, so more samples (best-of-k) reliably help, even if longer single chains stop helping.
  • Under a binary reward, the REINFORCE gradient reduces to rejection-sampling fine-tuning on successful paths, scaled by the success rate; longer chains emerge if successful paths tend to be longer.
  • Because only final answers are verified, intermediate tokens need not be faithful, and an imperfect verifier can enable reward hacking.

The core claim is that reasoning models are best understood as pathfinding through token space. Each sampled chain of thought is one path from the prompt x toward a set of correct answers S_x, where a verifier r(x, y) returns 1 for correct answers. The success probability p_x is the sum, over all paths that land in S_x, of the probability the model generates each path. Reframed this way, the usual intuition that "thinking longer means more likely correct" stops being the right question. What matters is the total probability mass on successful paths, and an individual chain need not converge to the right answer at all. That distinction is the author's proposed explanation for why forcing a trained model to think longer helps briefly and then plateaus or even degrades. The article then gives two ways to raise p_x. The first is best-of-k sampling: draw k paths and keep a successful one, which lifts success to 1 - (1 - p_x)^k. This improves monotonically with more samples as long as p_x > 0, though the scaling is over sample count rather than path length. The second is to change the model itself. The author shows that rejection-sampling fine-tuning (STaR is an early example) and RL with a binary verifiable reward reduce to nearly the same update. The expected reward equals p_x exactly, and the policy gradient estimate works out to the empirical success rate M/N times the average log-likelihood of the successful paths. In plain terms, a REINFORCE step is rejection-sampling fine-tuning on the model's own successes, weighted by how often it succeeds. This framing also accounts for the observation that chains of thought lengthen during RL. If successful paths happen to be longer, because they include backtracking, gather more environmental feedback, or simply got lucky, each update pulls the model toward longer paths. The article then connects this to amortization: RL spends compute searching for good paths up front so the trained model can produce them directly at inference time without repeated sampling. The author uses this to motivate later observations about unfaithful reasoning and reward hacking, since the model is optimizing for landing in S_x rather than for each step being a faithful account of its reasoning.

Ask about this article