Frontier reasoning models have shown remarkable performance improvements through reinforcement learning (RL), particularly in fields like mathematics and coding. However, this paper shifts focus from RL's benefits to the potential of base models alone. It explores whether these models can achieve comparable reasoning capabilities through a simple sampling method without additional training. The authors introduce an iterative sampling algorithm inspired by Markov chain Monte Carlo (MCMC) techniques. This approach leverages the base models' own likelihoods, yielding substantial improvements in reasoning performance across various tasks such as MATH500 and HumanEval, often matching or exceeding the results from RL-enhanced models. The study highlights that sampling directly from base models can maintain performance levels similar to RL without the pitfalls commonly associated with it, such as reduced diversity in generated outputs. Unlike RL methods, which require curated datasets and can suffer from training instabilities, the proposed algorithm operates without these constraints. The authors specifically benchmark their results against the Group Relative Policy Optimization (GRPO) algorithm, a leading RL approach, demonstrating that their method can outperform GRPO on out-of-domain tasks while maintaining high generation diversity across multiple samples. In their empirical evaluation, the algorithm is tested on various base models, including Qwen2.5-Math-7B and Phi-3.5-mini-instruct, revealing that existing base models possess greater single-shot reasoning capabilities than previously recognized. This work challenges the notion that RL is the only path to enhancing reasoning in LLMs and suggests that straightforward sampling techniques can tap into the untapped potential of base models. The findings imply a broader applicability of these sampling methods beyond the domains currently explored, indicating a significant shift in how we might approach reasoning tasks with LLMs.