Reasoning with Sampling: Your Base Model is Smarter Than You Think — part 3/3

Cleaned-text code points [31789, 43994). Bibliography omitted here; source.md includes it.

#### Diversity and pass@ k k performance.

Again, notice the peaked and highly concentrated likelihoods/confidences of GRPO relative to the distributional spread of power sampling in Figure 4 . This suggests GRPO exhibits a collapse in diversity while our sampler does not, aligning with the observation that RL-posttraining strongly sharpens the base model distribution at the expense of diversity ( Song et al., 2025 ) . To quantify the comparative diversity of power sampling relative to GRPO, we can plot the pass@ k k accuracy rate, where a question is solved if at least one of k k samples is accurate. Figure 5 shows exactly this: unlike GRPO, whose pass@ k k performance tapers off for large k k , power sampling strongly outperforms for k > 1 k>1 . Moreover, our performance curve supersedes that of the base model until finally converging in performance. In particular, we are able to achieve GRPO-level single-shot performance without compromising multi-shot performance (see Appendix A.2 for other domains), addressing a long-standing downside to RL-posttraining.

#### The effect of power distributions.

The two most important hyperparameters for power sampling are the choice of α \alpha and the number of MCMC (resampling) steps during sequence generation N MCMC N_{\text{MCMC}} . At the extremes, choosing α = 1.0 \alpha=1.0 samples from the base model directly, while taking α → ∞ \alpha\to\infty has the effect of deterministically accepting any resampled sequence that strictly increases the likelihood. Of course, even though higher base model likelihoods correlate with better reasoning (Figure 4 ), directly optimizing for likelihood is not necessarily optimal for reasoning, suggesting an ideal intermediate value of α \alpha .

In Figure 6 , we display MATH500 accuracies across various values of α \alpha and find that an intermediate α = 4.0 \alpha=4.0 outperforms other values, as expected. Noticeably, the accuracies of power sampling remain relatively stable beyond α ≥ 2.0 \alpha\geq 2.0 , suggesting that power sampling in practice is relatively robust to the choice of α \alpha .

#### Test-time scaling with MCMC steps.

On the other hand, N MCMC N_{\text{MCMC}} toggles the inference-time compute expended by our algorithm, providing a natural axis for test-time scaling. In Section 4.3 we raised the notion of a mixing time , or the number of MCMC steps required before adequately sampling from the target distribution. In our case, we expect that the fewer MCMC steps we take, the further our algorithm samples from the target p α p^{\alpha} .

We plot performance dependence on N MCMC N_{\text{MCMC}} in Figure 6 and notice a steady increase in accuracy until N MCMC = 10 N_{\text{MCMC}}=10 , beyond which accuracy remains roughly stable (not plotted). The accuracy difference from using fewer MCMC steps is noticeable but no more than 3 3 - 4 % 4\% between N MCMC = 2 N_{\text{MCMC}}=2 and N MCMC = 10 N_{\text{MCMC}}=10 . However, the jump in accuracy by using at least two steps as opposed to none is substantial ( 3 3 - 4 4 %).

We can even compute the total amount of tokens generated by our method relative to running GRPO. From ( 12 ), our sampler generates 1 4 ​ B ⋅ N MCMC ​ T \frac{1}{4B}\cdot{N_{\text{MCMC}}T} times as many tokens as standard inference to generate a sequence of length T T . Plugging in our experimental parameters N MCMC = 10 N_{\text{MCMC}}=10 , T = 679 T=679 (our average output length for MATH500) and B = 192 B=192 , running inference with power sampling incurs a multiplier of 8.84 × \textbf{8.84}\times the number of tokens as running standard inference. Since GRPO generates multiple rollouts per example during training, our method incurs roughly the same inference cost as one epoch of GRPO training , assuming 8 rollouts per sample with identical dataset sizes. Typically though, one GRPO epoch is still more expensive as it uses 16 rollouts and a training set that is larger than MATH500.

## 6 Conclusion

In this work, we present an algorithm that samples directly from a base model without any additional training or access to an external signal, achieving a single-shot reasoning performance that is on par with, and sometimes even better than, that of a state-of-the-art RL-posttraining algorithm. We use the discussion of RL distribution sharpening to motivate defining the power distribution as a valuable target distribution for reasoning. Although exact power distribution sampling is intractable, we employ classic MCMC techniques alongside the sequential structure of autoregressive generation to define our power sampling algorithm, which demonstrates strong empirical performance.

Our results suggest that base model capabilities are underutilized at sampling time and point towards a close relationship between high likelihood regions of the base model and strong reasoning capabilities. Employing additional compute at sampling-time with a stronger understanding of base model capabilites offers a promising direction for expanding the scope of reasoning beyond verifiability.

## 7 Acknowledgments

A.K. would like to thank the Paul and Daisy Soros Foundation, NDSEG Fellowship, and Kempner Institute for their support.

## Appendix A Appendix

### A.1 Additional Theoretical Discussion

In this section, we provide a stronger formalization of the phenomenon that power sampling downweights tokens that trap outputs in low-likelihood futures while low-temperature sampling does not.

###### Proposition 2 ( Informal ) .

Power sampling upweights tokens with small support but high likelihood completions, while low-temperature sampling upweights tokens with large support but low likelihood completions.

###### Definition 2 .

For the rest of this section, fix a prefix x 0 : t − 1 x_{0:t-1} . We say that x t x_{t} has marginal weight ε \varepsilon under the conditional next-token distribution if ∑ x > t p ⁡ ( x 0 , … , x t , … ​ x T ) = ε \sum_{x>t}p(x_{0},\dots,x_{t},\dots x_{T})=\varepsilon .

We consider a simplified model of the “critical window” or “pivotal token” phenomenon ( Li et al., 2025 ; Abdin et al., 2024 ) , which refers to intermediate tokens that strongly influence the quality of the final generation. We differentiate between pivotal tokens that lead to high-likelihood futures vs. low-likelihood ones.

###### Definition 3 .

At one extreme, a pivotal token maximally induces a high-likelihood completion if it places its entire marginal weight ε \varepsilon on one future (singular support); i.e., for only one choice of x > t x>t is p ⁡ ( x 0 , … , x t , … , x T ) p(x_{0},\dots,x_{t},\dots,x_{T}) nonzero. We call such a token a positive pivotal token.

###### Definition 4 .

At the other extreme, a pivotal token minimizes the likelihood of any future if its entire marginal weight ε \varepsilon is uniformly distributed across N N future completions. In other words, there exist N N completions x > t x>t such that p ⁡ ( x 0 , … , x t , … , x T ) p(x_{0},\dots,x_{t},\dots,x_{T}) are all nonzero with likelihood ε N \frac{\varepsilon}{N} . We call such a token a negative pivotal token.

Our simplified model of high and low-likelihood futures examines when positive pivotal tokens are favored over negative pivotal tokens under a given sampling distribution. In particular, we show that power sampling can upweight a positive pivotal token over a negative one even if the latter has a higher marginal weight, whereas low-temperature sampling always upweights the negative pivotal token in such a scenario.

Of course, whenever a positive pivotal token has higher marginal weight, both power sampling and low-temperature sampling will upweight it.

###### Proposition 3 .

Let x t x_{t} be a positive pivotal token with marginal weight ε \varepsilon , and let x t ′ x_{t}^{\prime} be a negative pivotal token with marginal weight ε ′ \varepsilon^{\prime} and support N N . Then if ε ′ N 1 − 1 / α < ε < ε ′ , \frac{\varepsilon^{\prime}}{N^{1-1/\alpha}}<\varepsilon<\varepsilon^{\prime}, (14) the future likelihood of x t x_{t} is higher than any future likelihood of x t ′ x_{t}^{\prime} . Moreover, power sampling upweights x t x_{t} over x t ′ x_{t}^{\prime} while low-temperature sampling upweights x t ′ x_{t}^{\prime} over x t x_{t} .

###### Proof.

Since α ≥ 1 \alpha\geq 1 , it follows that ε ′ N 1 − 1 / α > ε ′ N \frac{\varepsilon^{\prime}}{N^{1-1/\alpha}}>\frac{\varepsilon^{\prime}}{N} (15) and thus ε > ε ′ N \varepsilon>\frac{\varepsilon^{\prime}}{N} , establishing that the future completion likelihood of x t x_{t} is greater than that of x t ′ x_{t}^{\prime} (i.e. the assignment of positive and negative pivotal tokens is consistent).

Now, if ε < ε ′ \varepsilon<\varepsilon^{\prime} , then under the low-temperature distribution, the relative marginal weights on x t x_{t} and x t ′ x_{t}^{\prime} are ε α \varepsilon^{\alpha} and ε ′ α \varepsilon^{\prime\alpha} , so the probability of choosing x t x_{t} is downweighted relative to x t ′ x_{t}^{\prime} . However, for the power distribution, the relative marginal weights are p pow ​ ( x t | x < t ) = ε α p_{\text{pow}}(x_{t}|x_{<t})=\varepsilon^{\alpha} and p pow ​ ( x t ′ | x < t ) = ε ′ α N α − 1 p_{\text{pow}}(x_{t}^{\prime}|x_{<t})=\frac{\varepsilon^{\prime\alpha}}{N^{\alpha-1}} . Then, as long as ε α > ε ′ α N α − 1 ⇔ ε > ε ′ N 1 − 1 / α \varepsilon^{\alpha}>\frac{\varepsilon^{\prime\alpha}}{N^{\alpha-1}}\iff\varepsilon>\frac{\varepsilon^{\prime}}{N^{1-1/{\alpha}}} , token x t x_{t} will be upweighted relative to token x t ′ x_{t}^{\prime} .

In other words, the marginal weight on x t x_{t} can be less than the mass on x t ′ x_{t}^{\prime} under p p , but if the completion for x t x_{t} has higher likelihood than any individual completion for x t ′ x_{t}^{\prime} , power sampling favors x t x_{t} over x t ′ x_{t}^{\prime} .∎

### A.2 Pass@k Accuracies over Multiple Domains

In this section, we plot the pass@ k k performance of power sampling, GRPO, and the base model (Qwen2.5-Math-7B) over MATH500, GPQA, and HumanEval to demonstrate that our sampling algorithm is highly performant at both single-shot and multi-shot reasoning while maintaining response diversity. Power sampling is plotted with α = 4.0 \alpha=4.0 for MATH500 and GPQA and α = 1.67 \alpha=1.67 for HumanEval (this temperature exhibits slightly better results at earlier k k ). In all cases, both in-domain and out-of-domain for GRPO, power sampling has near universally better performance than both GRPO and the base model in pass@ k k for k > 1 k>1 and matches, if not exceeds, the base model upper bound at large k k .

One thing to note about these plots is that the loss in diversity varies noticeably from benchmark to benchmark. MATH500 and GPQA clearly show that GRPO has a significantly lower pass@ k k performance and diversity even for smaller k k , while on HumanEval GRPO exhibits better pass@ k k than the base model until k = 16 k=16 . We speculate this might be due to the fact that while MATH500 and GPQA are graded on a “singular” answer, HumanEval is based on completing code where multiple solutions might be accepted, indicating a reduction in diversity may emerge but at much higher k k .

### A.3 More Qualitative Examples

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .