Representation-Based Exploration for Language Models: From Test-Time to Post-Training — part 1/5
Cleaned-text code points [0, 15794). Bibliography omitted here; source.md includes it.
\declaretheorem [name=Theorem,parent=section]theorem \declaretheorem [name=Lemma,parent=section]lemma \declaretheorem [name=Assumption, parent=section]assumption \declaretheorem [name=Definition, parent=section]definition \declaretheorem [name=Condition, parent=section]condition \declaretheorem [name=Corollary, parent=section]corollary \declaretheorem [name=Claim, parent=section]claim \declaretheorem [qed= ⊲ \triangleleft ,name=Example,style=definition, parent=section]example \declaretheorem [name=Remark, parent=section]remark \declaretheorem [name=Proposition, parent=section]proposition \declaretheorem [name=Fact, parent=section]fact
# Representation-Based Exploration for Language Models: From Test-Time to Post-Training
###### Abstract
Reinforcement learning (RL) promises to expand the capabilities of language models, but it is unclear if current RL techniques promote the discovery of novel behaviors, or simply sharpen those already present in the base model. In this paper, we investigate the value of deliberate exploration—explicitly incentivizing the model to discover novel and diverse behaviors—and aim to understand how the knowledge in pre-trained models can guide this search. Our main finding is that exploration with a simple, principled, representation-based bonus derived from the pre-trained language model’s hidden states significantly improves diversity and pass@k rates—both for post-training, and in a novel inference-time scaling setting we introduce.
1. For inference-time, exploration with representation-based diversity improves efficiency, consistently improving pass@k rates across a variety of models and reasoning tasks. For example, for Qwen-2.5-14b-Instruct we obtain over 50% improvement in verifier efficiency on almost all tasks.
2. For post-training, we show that integrating this exploration strategy into an RL pipeline improves reasoning performance over that of the initial model and over standard RL post-training. For example, on AIME 2024 , our post-trained Qwen-2.5-7b-Instruct ’s pass@80 matches the pass@256 of GRPO on the same model, demonstrating a 3x improvement in test-time sample efficiency.
Overall, our findings suggest that deliberate exploration---with the right notion of diversity---is a practical path toward discovery of new behaviors beyond sharpening. 1 1 1 Website and code: https://rep-exp.github.io
## 1 Introduction
Reinforcement learning (RL) promises to endow agents with the ability to discover valuable behaviors autonomously, via closed-loop trial and error. For language modeling tasks with verifiable rewards, such as mathematical reasoning and code generation, post-training with reinforcement learning has already enabled impressive breakthroughs ( DeepSeek-AI, 2025 ; OpenAI, 2024 ) . Still, it is unclear whether contemporary RL implementations for language models attain the full promise of reinforcement learning. Rather than unlocking capabilities not present in the pre-trained model, there is increasing evidence ( Yue et al., 2025 ; Gandhi et al., 2025 ) that existing RL recipes ( Schulman et al., 2017 ; Rafailov et al., 2023 ; DeepSeek-AI, 2025 ) may simply amplify or sharpen ( Huang et al., 2025 ) behaviors that the base model can already execute, albeit with modest probability. While this can be mitigated through deliberate data curation and some algorithmic interventions ( He et al., 2025 ; Liu et al., 2025 ; Setlur et al., 2025 ) , data scale and quality are rapidly becoming bottlenecks, particularly in complex, open-ended domains where existing interventions fall short of eliciting desired behavior.
We argue that deliberate exploration—incentivizing the model to discover truly novel and diverse behavior—is an essential ingredient in realizing the full potential of RL for language model reasoning. Exploration has a rich history in both the theory and practice of RL, and exploration techniques tailored to deep networks ( Tang et al., 2017 ; Pathak et al., 2017 ; Burda et al., 2018 ; Osband et al., 2019 ) have received extensive investigation in the context of embodied decision making, including game playing and robotic control. These algorithms proceed from scratch, without pre-training, yet rapidly learn complex behaviors, demonstrating that they enable learning beyond the sharpening regime. If we can equip language models with exploration in a similar fashion, we may be able to advance reasoning capabilities without incurring exorbitant data curation costs.
In spite of the potential benefits of exploration, it is unclear which, if any, exploration technique from deep RL can be scaled to modern language models. A central challenge involves the scalable quantification of novelty and behavior diversity—and acting on this information—when the decision space under consideration is the combinatorially large space of language. At the same time, pre-trained language models contain tremendous prior knowledge compared to policies found in traditional embodied settings, which may be the key to guiding efficient exploration. This leads us to ask:
### 1.1 Contributions
Toward answering these questions, we focus on understanding whether exploration with diversity bonuses div ( x , y ) \texttt{div}(x,y) derived from a language model can effectively guide the search for diverse behaviors. We adopt a novel methodology ( Section 2 ) in which we first evaluate exploration in a simple, purely inference-time setting, then integrate our findings into post-training.
The inference-time selection problem ( Section 2 ). In this setting, we aim to select a small set of responses y 1 , … , y k y_{1},\ldots,y_{k} from a large set of candidates y 1 , … , y N y_{1},\ldots,y_{N} for a given prompt x x , such that the chosen set is as diverse as possible, and has high probability of including a positive response. This simple regime allows us to disentangle the role of diversity div ( x , y ) \texttt{div}(x,y) from other complex RL mechanisms, such as optimization and generalization.
Representation-based exploration improves diversity and efficiency. Our main finding is that exploration with a representation-based bonus ( Section 3 ) derived from the pre-trained language model’s hidden states significantly improves diversity and pass@k rates—both for our inference-time setting and for post-training. Our specific findings are as follows: 1. Inference-time ( Section 4 ). Inference-time exploration with representation-based diversity improves verifier efficiency. For example, we obtain over 50% improvement in verifier efficiency over standard sampling for Qwen-2.5-14b-Instruct on GSM8K , MATH , MBPP+ and Game-of-24 . See Fig. 1 for an overview of our results for this setting.
2. Post-training ( Section 5 ). Representation-based exploration can be incorporated into RL post-training, where its pass@ k k performance is competitive with both GRPO and the base model uniformly for all k k ( Fig. 2 ). Notably, representation-based exploration completely eliminates the “diversity collapse” phenomenon where RL degrades pass@ k k with respect to the base model for large k k ( Dang et al., 2025 ; Yue et al., 2025 ; Wu et al., 2025 ) . In addition, representation-based exploration induces responses that look much more novel under the base model ( Fig. 9 ).
Our findings, particularly these last two points, suggest that deliberate exploration is a practical path toward discovery of new behaviors beyond sharpening. Although our experiments focus on arguably the simplest principled representation-based exploration scheme—for which we already see substantial performance improvements—we expect that our two-pronged evaluation approach will enable a deeper understanding of the benefits and tradeoffs of more sophisticated strategies, which may help realize the full potential of reinforcement learning for language model reasoning.
#### Diversity-guided generation ( Section 4.2 )
As a proof of concept, we also evaluate an inference-time exploration algorithm that uses representation-based diversity to encourage exploration during the autoregressive generation process itself . We find that this improves pass@k for large k over naive sampling for Qwen-2.5-7b-Instruct on MATH .
## 2 Problem Setup: From Inference-Time to Post-Training
In this section, we describe the two problem settings we consider for exploration: inference-time selection and RL post-training. In what follows, π \pi denotes a language model that maps a prompt x ∈ 𝒳 x\in\mathcal{X} to a distribution over responses y ∈ 𝒴 y\in\mathcal{Y} , and r ⋆ ( x , y ) ∈ { 0 , 1 } r^{\star}(x,y)\in\{0,1\} denotes a verifiable reward function that measures correctness at a task of interest, such as whether the answer to a math question is correct, or whether a Python program passes unit tests.
#### Methodology and motivation
The goal of RL post-training is to find a policy π \pi that maximizes the expected reward 𝔼 y ∼ π ( ⋅ ∣ x ) [ r ⋆ ( x , y ) ] \En_{y\sim\pi(\cdot\mid{}x)}\left[r^{\star}(x,y)\right] . Given a budget k k of verifier queries per question at each data collection round, post-training algorithms such as GRPO ( Shao et al., 2024 ) update the model iteratively, where in each iteration they sample k k responses y 1 , … , y k ∼ i.i.d. π ( ⋅ ∣ x ) y_{1},\ldots,y_{k}\overset{\textrm{i.i.d.}}{\sim}\pi(\cdot\mid{}x) per prompt x x from the current model π \pi , query the verifier for a reward r ⋆ ( x , y i ) r^{\star}(x,y_{i}) for each response, and use observed rewards to update the model for the next iteration.
If the initial model π \pi has poor support over rewarding behavior—i.e., if r ⋆ ( x , y ) = 0 r^{\star}(x,y)=0 , with high probability under y ∼ π ( ⋅ ∣ x ) y\sim\pi(\cdot\mid{}x) —common RL algorithms such GRPO or PPO ( Schulman et al., 2017 ) will not make any progress. This motivates interventions for exploration such as bonuses ( Tang et al., 2017 ; Pathak et al., 2017 ; Burda et al., 2018 ; Osband et al., 2019 ) and alternative sampling strategies ( Holtzman et al., 2020 ; Minh et al., 2025 ) . However, understanding the benefits and tradeoffs of these interventions in RL post-training is challenging because exploration interacts with optimization and generalization. To isolate exploration from these other considerations, we center our investigation around a task we refer to as inference-time selection , validating interventions in this setting before integrating them into the RL post-training pipeline.
### 2.1 Inference-time selection
In the inference-time selection problem, we aim to use a fixed model π \pi to build a set of k k responses to a given prompt x x that are maximally diverse and have high probability of containing a positive response. As a simple baseline, we may independently sample k k responses from the model—potentially with high-temperature sampling, nucleus or min-p sampling, or other modified sampling schemes. However, the limitations of these baselines are (i) they may not effectively capture the model’s understanding of diversity, and (ii) by sampling independently, we may waste verifier queries on redundant responses.
Instead, we focus on selection-based approaches that initially sample a large set of candidate responses to the prompt, then use a diversity bonus div ( x , y ) \texttt{div}(x,y) derived from the model to filter this set down to a smaller, more diverse “coreset” ( Clarkson, 2010 ; Feldman et al., 2020 ) that is passed to the verifier. Formally, we consider the following protocol: For each prompt x x , we (1) sample an initial batch of N N responses y 1 , … , y N ∼ π ( ⋅ ∣ x ) y_{1},\ldots,y_{N}\sim\pi(\cdot\mid{}x) , (2) use the inference-time selection algorithm 𝙰𝚕𝚐 \mathtt{Alg} to select k k of these responses (a subset S ⊂ [ N ] S\subset[N] of size | S | = k |S|=k ), and (3) query the verifier and record if any of the selected responses are rewarding. That is, we measure pass@ k k , 𝔼 y 1 , … , y N ∼ π ( ⋅ ∣ x ) [ 𝔼 S ∼ 𝙰𝚕𝚐 ( x , y 1 , … , y N ) [ max i ∈ S [ r ⋆ ( x , y i ) ] ] ] . \displaystyle\En_{y_{1},\ldots,y_{N}\sim\pi(\cdot\mid{}x)}\big[\En_{S\sim\mathtt{Alg}(x,y_{1},\ldots,y_{N})}\big[\max_{i\in S}[r^{\star}(x,y_{i})]\big]\big]. (1) Importantly, the filtering algorithm operates without the verifier, and so successfully retaining high-quality responses translates to improved verifier efficiency (i.e., number of responses for which we query the verifier) over the initial set of responses. Thus, a useful diversity bonus div ( x , y ) \texttt{div}(x,y) should yield a coreset that is maximally “exploratory,” in the sense that it is the most diverse set of responses that can be selected for a fixed budget of verifier queries. For example, in math reasoning settings, we would like to select the distinct-but-plausible proof strategies for a given problem, thus covering the space of potential proofs and maximizing the chance of selecting a correct one.
{remark} While we mainly introduce inference-time selection as a stepping stone to post-training (i.e., algorithms in this setting are not more compute-efficient than naive sampling, even if they are more verifier-efficient), we do expect inference-time exploration to be useful in its own right for domains where querying a verifier is costly or difficult (e.g., collecting feedback from expert-level annotators), allowing for more sample- and hence cost-efficient data curation. For preliminary results in one such domain, please refer to Section 4.3 .
### 2.2 Reinforcement learning post-training
As described earlier, RL post-training (e.g., with GRPO or PPO) proceeds by iteratively sampling batches of responses, querying the verifier, and using the feedback to update the current policy. After selecting a checkpoint π ^ \hat{\pi} , we evaluate performance via pass@ k k under standard generation, 𝔼 y 1 , … , y k ∼ π ^ ( ⋅ ∣ x ) [ max i ∈ [ k ] r ⋆ ( x , y i ) ] \En_{y_{1},\ldots,y_{k}\sim\hat{\pi}(\cdot\mid{}x)}\big[\max_{i\in[k]}r^{\star}(x,y_{i})\big] . There are two natural approaches to integrate exploration methods into this process. The first is to adjust the independent sampling process only (e.g., through nucleus sampling or min-p sampling), and the second is to augment the training objective with an exploration bonus div ( x , y ) \texttt{div}(x,y) . Our experiments focus on the latter approach; however, based on our results for inference-time selection, we expect that incorporating representation-based exploration into the sampling process will also improve RL post-training performance. Indeed, our two-pronged evaluation is motivated by the hypothesis that diversity bonuses div ( x , y ) \texttt{div}(x,y) that perform well at inference-time also perform well in post-training.
## 3 Representation-Based Exploration: Inference-Time and RL
Having motivated our setup, we now turn to the question of what diversity bonuses div ( x , y ) \texttt{div}(x,y) are suitable for exploration with language models. While many metrics have been proposed in the literature ( Tang et al., 2017 ; Pathak et al., 2017 ; Burda et al., 2018 ; Osband et al., 2019 ) , the challenge in adapting these techniques to language models is to simultaneously (i) capture the model’s understanding and (ii) allow for efficient computation at scale. For example, count-based exploration ( Tang et al., 2017 ) is simple, but unsuited to large decision spaces. On the other hand, approaches based on intrinsic curiosity ( Pathak et al., 2017 ) , random network distillation ( Burda et al., 2018 ) , and posterior sampling ( Osband et al., 2019 ) are better suited to large or continuous spaces, but require additional learning machinery (i.e., auxiliary networks), which introduces significant complexity when scaling to language models.