Reasoning with Sampling: Your Base Model is Smarter Than You Think — part 1/3

Cleaned-text code points [0, 15823). Bibliography omitted here; source.md includes it.

# Reasoning with Sampling: Your Base Model is Smarter Than You Think

###### Abstract

Frontier reasoning models have exhibited incredible capabilities across a wide array of disciplines, driven by posttraining large language models (LLMs) with reinforcement learning (RL). However, despite the widespread success of this paradigm, much of the literature has been devoted to disentangling truly novel behaviors that emerge during RL but are not present in the base models. In our work, we approach this question from a different angle, instead asking whether comparable reasoning capabilites can be elicited from base models at inference time by pure sampling , without any additional training . Inspired by Markov chain Monte Carlo (MCMC) techniques for sampling from sharpened distributions, we propose a simple iterative sampling algorithm leveraging the base models’ own likelihoods. Over different base models, we show that our algorithm offers substantial boosts in reasoning that nearly match and even outperform those from RL on a wide variety of single-shot tasks, including MATH500, HumanEval, and GPQA. Moreover, our sampler avoids the collapse in diversity over multiple samples that is characteristic of RL-posttraining. Crucially, our method does not require training, curated datasets, or a verifier, suggesting broad applicability beyond easily verifiable domains.

## 1 Introduction

Reinforcement learning (RL) has become the dominant paradigm for enhancing the reasoning capabilities of large language models (LLMs) ( Guo et al., 2025 ; Hu et al., 2025 ) . Equipped with a reward signal that is typically automatically verifiable, popular RL techniques have been successfully applied to posttrain frontier models, leading to sizeable performance gains in domains like math, coding, and science ( Hendrycks et al., 2021 ; Li et al., 2022 ; Rein et al., 2024 ) .

Despite the widespread empirical success of RL for LLMs, a large body of literature has centered around the following question: are the capabilities that emerge during RL-posttraining fundamentally novel behaviors that are not present in the base models? This is the question of distribution sharpening ( He et al., 2025 ; Shao et al., 2025 ; Yue et al., 2025 ) : that is, whether the posttrained distribution is simply a “sharper” version of the base model distribution, instead of placing mass on reasoning traces the base model is unlikely to generate.

Several works point towards the difficulty in learning new capabilities with RL-posttraining. He et al. (2025) ; Song et al. (2025) compare the pass@ k k (multi-shot) scores of base models with posttrained models, finding that for large k k , base models actually outperform while the latter suffer from degraded generation diversity. In such cases, RL appears to redistribute pass@ k k performance to single-shot performance at the expense of multi-shot reasoning. Yue et al. (2025) also notes that the reasoning traces post-RL are tightly concentrated at high likelihoods/confidences under the base model, seemingly drawing from existing high-likelihood capabilities. We illustrate this point in our own experiments in Figure 4 . Regardless, the advantage of RL-posttraining for single-shot reasoning has remained, as of yet, undeniable.

In this paper, we present a surprising result: sampling directly from the base model can achieve single-shot reasoning capabilites on par with those from RL .

We propose a sampling algorithm for base models that leverages additional compute at inference time, achieving single-shot performance that nearly matches RL-posttraining on in-domain reasoning tasks and can even outperform on out-of-domain reasoning tasks. Furthermore, we observe that generation diversity does not degrade with our sampler; in fact, our pass@ k k (multi-shot) performance strongly outperforms RL. We benchmark specifically against Group Relative Policy Optimization (GRPO), which is the standard RL algorithm for enhancing LLM reasoning ( Shao et al., 2024 ) .

Crucially, our algorithm is training-free , dataset-free , and verifier-free , avoiding some of the inherent weaknesses of RL methods including extensive hyperparameter sweeps to avoid training instabilities, the need to curate a diverse and expansive posttraining dataset, and the lack of guaranteed access to a ground truth verifier/reward signal ( Prabhudesai et al., 2025 ) .

Our contributions can be summarized as follows: i) We introduce the power distribution as a useful sampling target for reasoning tasks. Since it can be explicitly specified with a base LLM, no additional training is required.

ii) We further introduce an approximate sampling algorithm for the power distribution using a Markov chain Monte Carlo (MCMC) algorithm that iteratively resamples token subsequences according to their base model likelihoods.

iii) We empirically demonstrate the effectiveness of our algorithm over a range of models (Qwen2.5-Math-7B, Qwen2.5-7B, Phi-3.5-mini-instruct) and reasoning tasks (MATH500, HumanEval, GPQA, AlpacaEval 2.0). Our results show that sampling directly from the base model can achieve results on par with GRPO. In fact, for some out-of-domain tasks, our algorithm consistently outperforms the RL baseline. Moreover, over multiple samples, we avoid the collapse in diversity afflicting RL-posttraining, achieving the best of both worlds in terms of single-to-few-shot reasoning capabilities as well as sample diversity.

Our results collectively illustrate that existing base models are much more capable at single-shot reasoning than current sampling methods reveal.

## 2 Related Works

#### Reinforcement learning for LLMs.

RL has been instrumental in posttraining LLMs. Early on, RL with human feedback (RLHF) ( Ouyang et al., 2022 ) was developed as a technique to align LLMs with human preferences using a trained reward model. Recently, RL with verifiable rewards (RLVR) has emerged as a powerful new posttraining technique, where many works ( Guo et al., 2025 ; Lambert et al., 2024 ; Hu et al., 2025 ; Zeng et al., 2025 ) discovered that a simple, end-of-generation reward given by an automated verifier could substantially enhance performance on difficult reasoning tasks in mathematics and coding. The Group Relative Policy Optimization (GRPO) algorithm was at the center of these advances ( Shao et al., 2024 ) . Building off of this success, many subsequent works have examined using reward signals derived from internal signals such as self-entropy ( Zhao et al., 2025 ) , confidence ( Prabhudesai et al., 2025 ) , and even random rewards ( Shao et al., 2025 ) . Similar to these works, our paper examines base model likelihoods as a mechanism for improving reasoning performance, but crucially, our technique is training-free .

#### Autoregressive MCMC sampling with LLMs.

Prior works have explored integrating classic MCMC techniques with autoregressive sampling. Many settings including red-teaming, prompt-engineering, and personalized generation can be framed as targeting sampling from the base LLM distribution but tilted towards an external reward function. Zhao et al. (2024) proposes learning intermediate value functions that are used in a Sequential Monte Carlo (SMC) framework ( Chopin, 2004 ) , where multiple candidate sequences are maintained and updated according to their expected future reward. Similarly, Faria et al. (2024) proposes a Metropolis-Hastings (MH) algorithm, which instead of maintaining multiple candidates performs iterative resampling, again updating according to expected reward. Methodologically, our sampling algorithm is most similar to this latter work, but the crucial difference is that our target sampling distribution is completely specified by the base LLM, avoiding the need for an external reward .

#### Annealed sampling for diffusion.

In the statistical physics and Monte Carlo literature, sampling from p α p^{\alpha} is known as sampling from an annealed , or tempered , distribution ( Neal, 1998 ) and has inspired a new wave of interest within the diffusion community. Indeed, in traditional MCMC sampling, annealing is used as a way to avoid mode-collapse during sampling and more accurately sample from complex multimodal distributions ( Łatuszyński et al., 2025 ) . This has re-emerged as inference-time sampling methods for diffusion that aim to steer a pretrained model towards “tilted distributions” ( Du et al., 2023 ; Kim et al., 2025 ; Karan et al., 2025 ; Wang et al., 2025 ; Kong et al., 2025 ; Zhang et al., 2025 ) . Where traditional RL techniques exhibit mode collapse, applications in the physical sciences ( Sambridge, 2014 ) require multimodal sampling. To this end, works such as Du et al. (2023) ; Wang et al. (2025) ; Kim et al. (2025) construct sequences of annealed distributions to ease the transition from base diffusion distribution to tilted distribution. Other works ( Skreta et al., 2025 ; Xu et al., 2025 ) intentionally target sampling from p α p^{\alpha} for α > 1 \alpha>1 as a means of generating higher quality samples from the base diffusion model, which is particularly popular for generating more designable proteins ( Geffner et al., 2025 ) .

## 3 Preliminaries

Let 𝒳 \mathcal{X} be a finite vocabulary of tokens, and let 𝒳 T \mathcal{X}^{T} denote the set of finite sequences of tokens x 0 : T = ( x 0 , x 1 , … , x T ) x_{0:T}=(x_{0},x_{1},\dots,x_{T}) , where x i ∈ 𝒳 x_{i}\in\mathcal{X} for all i i and T ∈ ℤ ≥ 0 T\in\mathbb{Z}_{\geq 0} is some nonnegative integer. For convenience, for a given t t , let x < t = ( x 0 , … , x t − 1 ) x_{<t}=(x_{0},\dots,x_{t-1}) and x > t = ( x t + 1 , … , x T ) x_{>t}=(x_{t+1},\dots,x_{T}) , with similar definitions for x ≤ t x_{\leq t} and x ≥ t x_{\geq t} . In general, 𝐱 \mathbf{x} refers to a token sequence x 0 : T x_{0:T} , where T T is implicitly given.

Then an LLM defines a distribution p p over token sequences 𝒳 T \mathcal{X}^{T} by autoregressively learning the conditional token distributions p ⁡ ( x t | x < t ) p(x_{t}|x_{<t}) for all t t , giving the joint distribution via the identity p ( x 0 : T ) = ∏ t = 0 T p ( x t | x < t ) . p(x_{0:T})=\prod_{t=0}^{T}p(x_{t}|x_{<t}). (1)

To sample a sequence from p p , we simply sample from the LLM token by token using the conditional distributions, which by ( 1 ) directly samples from the joint distribution.

## 4 MCMC Sampling for Power Distributions

In this section, we introduce our sampling algorithm for base models. Our core intuition is derived from the notion of distribution sharpening posed in Section 1 . Sharpening a reference distribution refers to reweighting the distribution so that high likelihood regions are further upweighted while low likelihood regions are downweighted, biasing samples heavily towards higher likelihoods under the reference. Then if RL posttrained models really are just sharpened versions of the base model, we should be able to explicitly specify a target sampling distribution that achieves the same effect.

We organize this section as follows. Section 4.1 presents this target sharpened distribution and provides some mathematical motivation for why its samples are amenable for reasoning tasks. Section 4.2 introduces a general class of Markov chain Monte Carlo (MCMC) algorithms aimed at actually sampling from this target distribution, and finally, Section 4.3 details our specific implementation for LLMs.

### 4.1 Reasoning with Power Distributions

One natural way to sharpen a distribution p p is to sample from the power distribution p α p^{\alpha} . Since p ⁡ ( 𝐱 ) > p ⁡ ( 𝐱 ′ ) ⟹ p ​ ( 𝐱 ) α p ​ ( 𝐱 ′ ) α > p ⁡ ( 𝐱 ) p ⁡ ( 𝐱 ′ ) ( α ∈ [ 1 , ∞ ] ) , p(\mathbf{x})>p(\mathbf{x^{\prime}})\implies\frac{p(\mathbf{x})^{\alpha}}{p(\mathbf{x^{\prime}})^{\alpha}}>\frac{p(\mathbf{x})}{p(\mathbf{x^{\prime}})}\qquad(\alpha\in[1,\infty]), (2) it follows that exponentiating p p increases the relative weight on higher likelihood sequences ( 𝐱 \mathbf{x} ) while decreasing the relative weight on lower likelihood ones ( 𝐱 ′ \mathbf{x^{\prime}} ) (see Figure 2 for a visualization).

A related but well-known sharpening strategy is low-temperature sampling ( Wang et al., 2020 ) , which exponentiates the conditional next-token distributions at each step: p temp ​ ( x t | x 0 ​ … ​ x t − 1 ) = p ​ ( x t | x t − 1 ​ … ​ x 0 ) α ∑ x t ′ ∈ 𝒳 p ​ ( x t ′ | x t − 1 ​ … ​ x 0 ) α , p_{\text{temp}}(x_{t}|x_{0}\dots x_{t-1})=\frac{p(x_{t}|x_{t-1}\dots x_{0})^{\alpha}}{\sum_{x_{t}^{\prime}\in\mathcal{X}}p(x_{t}^{\prime}|x_{t-1}\dots x_{0})^{\alpha}}, (3) where the temperature is τ = 1 / α \tau=1/\alpha . A common misconception is that sampling with ( 3 ) over T T tokens is equivalent to sampling from p α p^{\alpha} ; however, this is false in a subtle yet crucial way, as we illuminate in the following.

###### Proposition 1 .

Low-temperature sampling does not sample from the power distribution p α p^{\alpha} .

###### Proof.

We show that the associated conditional next-token distributions are distinct at each timestep t t . The conditional distribution on x t x_{t} for p α p^{\alpha} is given by p pow ​ ( x t | x 0 ​ … ​ x t − 1 ) = ∑ x > t p ​ ( x 0 , … , x t , … , x T ) α ∑ x ≥ t p ​ ( x 0 , … , x t , … , x T ) α . p_{\text{pow}}(x_{t}|x_{0}\dots x_{t-1})=\frac{\sum_{x_{>t}}p(x_{0},\dots,x_{t},\dots,x_{T})^{\alpha}}{\sum_{x_{\geq t}}p(x_{0},\dots,x_{t},\dots,x_{T})^{\alpha}}. (4)

Using Bayes rule p ⁡ ( x t | x t − 1 ​ … ​ x 0 ) = p ⁡ ( x 0 , … , x t ) p ⁡ ( x 0 , … , x t − 1 ) = ∑ x > t p ⁡ ( x 0 , … , x t , … , x T ) ∑ x ≥ t p ⁡ ( x 0 , … , x t , … , x T ) , p(x_{t}|x_{t-1}\dots x_{0})=\frac{p(x_{0},\dots,x_{t})}{p(x_{0},\dots,x_{t-1})}=\frac{\sum_{x_{>t}}p(x_{0},\dots,x_{t},\dots,x_{T})}{\sum_{x_{\geq t}}p(x_{0},\dots,x_{t},\dots,x_{T})}, (5) we can rewrite the low-temperature marginal ( 3 ) as p temp ​ ( x t | x 0 ​ … ​ x t − 1 ) = ( ∑ x > t p ⁡ ( x 0 , … , x t , … , x T ) ) α ∑ x t ′ ( ∑ x > t p ⁡ ( x 0 , … , x t , … , x T ) ) α . p_{\text{temp}}(x_{t}|x_{0}\dots x_{t-1})=\frac{\left(\sum_{x_{>t}}p(x_{0},\dots,x_{t},\dots,x_{T})\right)^{\alpha}}{\sum_{x_{t}^{\prime}}\left(\sum_{x_{>t}}p(x_{0},\dots,x_{t},\dots,x_{T})\right)^{\alpha}}. (6)

Ignoring normalizations for clarity, the relative weight on token x t x_{t} for sampling from p α p^{\alpha} is given by a sum of exponents p pow ​ ( x t | x < t ) ∝ ∑ x > t p ​ ( x 0 , … , x t , … , x T ) α . p_{\text{pow}}(x_{t}|x_{<t})\propto\sum_{x_{>t}}p(x_{0},\dots,x_{t},\dots,x_{T})^{\alpha}. (7) Meanwhile, the relative weight for low-temperature sampling is given by an exponent of sums p temp ​ ( x t | x < t ) ∝ ( ∑ x > t p ⁡ ( x 0 , … , x t , … , x T ) ) α . p_{\text{temp}}(x_{t}|x_{<t})\propto\left(\sum_{x_{>t}}p(x_{0},\dots,x_{t},\dots,x_{T})\right)^{\alpha}. (8) Since the relative weights of next-token prediction are distinct for each sampling strategy, it follows that the joint distribution over seqeunces must also be distinct for each sampler. Hence, the distribution on sequences given by low-temperature sampling is not the same as the one given by p α p^{\alpha} . ∎

One intuitive way to understand this difference is that low-temperature sampling does not account for how exponentiation sharpens the likelihoods of “future paths” at time step t t , instead “greedily” averaging all these future likelihoods ( exponent of sums ( 8 )). On the other hand, sampling from p α p^{\alpha} inherently accounts for future completions as it exponentiates all future paths ( sum of exponents ( 7 )) before computing the weights for next-token prediction. This has the following consequence:

###### Observation 1 .

The power distribution upweights tokens with few but high likelihood future paths, while low-temperature sampling upweights tokens with several but low likelihood completions.

###### Example 1 .