CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models — part 1/4

Cleaned-text code points [0, 15014). Bibliography omitted here; source.md includes it.

# CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models

###### Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for enhancing the reasoning ability of Large Language Models (LLMs). Yet current RLVR methods often explore poorly, leading to premature convergence and entropy collapse. To address this challenge, we introduce Curiosity-Driven Exploration (CDE) , a framework that leverages the model’s own intrinsic sense of curiosity to guide exploration. We formalize curiosity with signals from both the actor and the critic: for the actor, we use perplexity over its generated response, and for the critic, we use the variance of value estimates from a multi-head architecture. Both signals serve as an exploration bonus within the RLVR framework to guide the model. Our theoretical analysis shows that the actor-wise bonus inherently penalizes overconfident errors and promotes diversity among correct responses; moreover, we connect the critic-wise bonus to the well-established count-based exploration bonus in RL. Empirically, our method achieves an approximate +3 point improvement over standard RLVR using GRPO/PPO on AIME benchmarks. Further analysis identifies a calibration collapse mechanism within RLVR, shedding light on common LLM failure modes.

## 1 Introduction

The reasoning ability of Large Language Models (LLMs) has achieved remarkable performance across diverse application domains such as mathematics ( Shao et al., 2024 ) and coding ( Guo et al., 2024 ) . A central challenge in this development is how to efficiently elicit high-quality Chain-of-Thought (CoT) reasoning. A major breakthrough earlier this year was the introduction of reinforcement learning with verifiable rewards (RLVR), a training paradigm in which models are optimized directly using the signal of final-answer correctness. This approach removes the burden of designing and training potentially fragile reward models. Despite the emergence of various RLVR training algorithms, such as GRPO ( Guo et al., 2024 ) and DAPO ( Yu et al., 2025 ) , key issues remain. In particular, problems such as premature convergence and phenomena like entropy collapse ( Cui et al., 2025 ) have been widely observed during training, posing fundamental challenges to the stability and effectiveness of RLVR.

These challenges stem from the classic exploration-exploitation dilemma in reinforcement learning ( Sutton & Barto, 2018 ) . Phenomena like entropy collapse reveal a critical flaw in the training process: it is heavily biased towards exploitation, causing models to converge prematurely instead of sufficiently exploring their environment for better solutions. Although the RL literature encompasses a wide range of exploration strategies, these methods exhibit significant limitations when applied to LLMs. Simple heuristics, including entropy bonuses ( Haarnoja et al., 2018 ) and ϵ \epsilon -greedy policies ( Sutton & Barto, 2018 ) , either injecting randomness to the environment or encouraging the policy to be more stochastic. These methods are either provably suboptimal in theory ( Dann et al., 2022 ) or demonstrate debatable effectiveness in complex environments like Deep RL ( Andrychowicz et al., 2021 ) and LLM-based reasoning ( Cui et al., 2025 ; Shen, 2025 ) . More principled methods are count-based, which incentivize visiting rarely explored state–action pairs. Algorithms such as UCB ( Lai, 1987 ) for multi-armed bandits, LinUCB ( Li et al., 2010 ) , LSVI-UCB ( Jin et al., 2020 ) for linear bandits/MDPs achieve near-optimal exploration guarantees across a variety of settings. However, these methods (i) require computationally intensive operations such as matrix inversion, and (ii) heavily depend on highly expressive representations of state-action pair (reasoning paths), which become impractical for reasoning-focused LLMs with long chains of thought. Thus, developing efficient and scalable exploration methods for LLMs remains a key open challenge.

In our preliminary experiments, we investigate the direct application of count-based exploration methods to the RLVR setting. To avoid the computational burden of matrix inversion, we adopt the SimHash technique ( Tang et al., 2017 ) , which maps the embedding of a CoT response into a discrete hash code, and then uses the visitation frequency of hash cells as pseudo-counts (see Section 3.1 for details). However, as illustrated in Figure 1 , this approach proves problematic: it is difficult to meaningfully characterize a complex CoT reasoning trajectory with a fixed embedding vector. In practice, most responses collapse into the same or neighboring hash grids, leading to a highly concentrated distribution of counts and thus undermining the effectiveness of count-based exploration for RLVR.

In this work, we propose an intuitive approach that leverages the model’s intrinsic sense of curiosity as a guide for exploration. An LLM, having been trained on vast reasoning corpora, develops a sophisticated internal model of what constitutes a familiar versus a novel reasoning pattern. This parallels early childhood development ( Chu & Schulz, 2020 ) , where learning is not driven by a external summary and count of experiences, but is instead propelled by an intrinsic curiosity to explore novel situations. We formalize this principle in our Curiosity-Driven Exploration (CDE) framework, which considers curiosity signals from both the actor and the critic. For the actor, perplexity (PPL) over its generated response serves as the curiosity measure. For the critic, we measure curiosity via the variance of its posterior value distribution. We then approximate this posterior by extending the PPO framework with a multi-head, bootstrapped structure. The curiosity signals are served as an exploration bonus, shaping the reward and advantage functions to effectively guide exploration.

Our theoretical analysis offers further insights into the properties of our method. (i) Theorem 3.1 interprets the proposed perplexity-based bonus, showing that it intrinsically penalizes overconfident errors while encouraging diversity among correct responses. (ii) Theorem 3.2 establishes that in the linear MDP setting, our critic-based exploration bonus is theoretically equivalent to classical count-based bonuses, grounding our approach in established exploration principles.

Our empirical evaluation demonstrates consistent performance gains across four widely used mathematics benchmarks (AIME25, AIME24, AMC23, and MATH), including an approximate +3 point improvement on the challenging AIME benchmarks. Furthermore, our analysis of the training process supports our theoretical findings and reveals a phenomenon we term calibration collapse : under a naive GRPO policy, the model’s confidence progressively decouples from its correctness, while adding PPL bonus mitigates this miscalculation.

## 2 Preliminaries: RLVR, GRPO and PPO

We formulate the language generation process of LLMs as a sequential decision-making problem ( Yu et al., 2025 ; Yue et al., 2025 ) . Specifically, we consider two reinforcement learning algorithms: Group Relative Policy Optimization (GRPO), a critic-free method, and Proximal Policy Optimization (PPO), a canonical actor–critic method. We adopt the training paradigm of Reinforcement Learning with Verifiable Rewards (RLVR) ( Guo et al., 2025 ; Lambert et al., 2024 ) and utilize a rule-based verifier to compare the generated response with the ground truth to judge its correctness.

### 2.1 Group Relative Policy Optimization (GRPO, Shao et al. 2024 )

GRPO is an REINFORCE-style optimization algorithm. Let π θ \pi_{\theta} denote the LLM policy with parameters θ \theta . At each training step, given a prompt q q sampled from the dataset 𝒟 \mathcal{D} , the current policy π θ \pi_{\theta} generates a group of G G candidate outputs { o 1 , o 2 , … , o G } \{o_{1},o_{2},\ldots,o_{G}\} . For each candidate o i o_{i} , we compute its total reward r i = r ⁡ ( o i , q ) r_{i}=r(o_{i},q) .

The advantage for each output is computed by normalizing its reward with respect to the group’s rewards: A i = r i − mean ⁡ ( r 1 , … , r G ) std ⁡ ( r 1 , … , r G ) + δ , A_{i}=\frac{r_{i}-\operatorname{mean}(r_{1},\ldots,r_{G})}{\operatorname{std}(r_{1},\ldots,r_{G})+\delta}, where δ \delta is a small constant for numerical stability. The same advantage A i A_{i} is applied to all tokens in o i o_{i} . Let π θ old \pi_{\theta_{\text{old}}} be the policy from the previous step and π ref \pi_{\text{ref}} the original pre-trained model. GRPO maximizes: ℒ GRPO ( θ ) = 𝔼 q ∼ 𝒟 , { o i } ∼ π θ old [ 1 G ∑ i = 1 G 1 | o i | ∑ t = 1 | o i | ℒ θ ( r ~ i , t , A i ) ] − β D KL ( π θ ∥ π ref ) , \mathcal{L}_{\text{GRPO}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\{o_{i}\}\sim\pi_{\theta_{\text{old}}}}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\mathcal{L}_{\theta}(\tilde{r}_{i,t},A_{i})\right]-\beta D_{\text{KL}}\left(\pi_{\theta}\|\pi_{\text{ref}}\right), where the clipped objective is ℒ θ ​ ( r ~ i , t , A i ) = min ⁡ ( r ~ i , t ​ A i , clip ​ ( r ~ i , t , 1 − ε , 1 + ε ) ​ A i ) , r ~ i , t = π θ ​ ( o i , t ∣ q , o i , < t ) π θ old ​ ( o i , t ∣ q , o i , < t ) . \mathcal{L}_{\theta}(\tilde{r}_{i,t},A_{i})=\min\left(\tilde{r}_{i,t}A_{i},\text{clip}(\tilde{r}_{i,t},1-\varepsilon,1+\varepsilon)A_{i}\right),\quad\tilde{r}_{i,t}=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})}. Here, ε \varepsilon and β \beta control the ratio clipping threshold and the KL-penalty strength, respectively. The clipping mitigates large, unstable policy updates, while the KL term constrains deviation from π ref \pi_{\text{ref}} .

### 2.2 Proximal Policy Optimization (PPO, Schulman et al. 2017 )

PPO is an actor–critic algorithm that maintains both a policy (actor) π θ \pi_{\theta} and a value function (critic) V ϕ V_{\phi} with parameters ϕ \phi , estimating the expected total reward from a given state (prompt and sequence prefix). The advantage function in PPO leverages the critic to reduce variance. Specifically, Generalized Advantage Estimation (GAE) is applied to compute token-level advantages. For an output o i o_{i} with sentence-level reward r i r_{i} , the GAE at token t t is: A i , t = ∑ l = t | o i | ( γ ​ λ ) l − t ​ δ i , l , A_{i,t}=\sum_{l=t}^{|o_{i}|}(\gamma\lambda)^{l-t}\delta_{i,l}, where δ i , l = r i , l + γ ​ V ϕ ​ ( q , o i , ≤ l + 1 ) − V ϕ ​ ( q , o i , ≤ l ) , \delta_{i,l}=r_{i,l}+\gamma V_{\phi}(q,o_{i,\leq l+1})-V_{\phi}(q,o_{i,\leq l}), and in our setting r i , l = 0 r_{i,l}=0 for all non-terminal tokens, with r i , | o i | = r i r_{i,|o_{i}|}=r_{i} . The hyperparameters γ \gamma and λ \lambda are the discount factor and GAE trace-decay, respectively. The PPO objective is: ℒ PPO ( θ , ϕ ) = 𝔼 q ∼ 𝒟 , { o i } ∼ π θ old [ 1 | o i | ∑ t = 1 | o i | [ ℒ θ ( r ~ i , t , A i , t ) − c 1 ℒ ϕ ( q , o i , < t , r i ) ] ] − β D KL ( π θ ∥ π ref ) , \mathcal{L}_{\text{PPO}}(\theta,\phi)=\mathbb{E}_{q\sim\mathcal{D},\{o_{i}\}\sim\pi_{\theta_{\text{old}}}}\Big[\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\left[\mathcal{L}_{\theta}(\tilde{r}_{i,t},A_{i,t})-c_{1}\mathcal{L}_{\phi}(q,o_{i,<t},r_{i})\right]\Big]-\beta D_{\text{KL}}\left(\pi_{\theta}\|\pi_{\text{ref}}\right), where ℒ θ \mathcal{L}_{\theta} is as in GRPO but with per-token A i , t A_{i,t} , and the value loss is: ℒ ϕ ​ ( ϕ ) = ( V ϕ ​ ( q , o i , < t ) − r i ) 2 . \mathcal{L}_{\phi}(\phi)=\left(V_{\phi}(q,o_{i,<t})-r_{i}\right)^{2}. In practice, we alternate optimization of the actor ( θ \theta ) and the critic ( ϕ \phi ).

## 3 CDE: Curiosity-Driven Exploration

In this section, we first explore count-based exploration and identify two of its key challenges. To overcome these challenges, we propose Curiosity-Driven Exploration (CDE), a systematic framework that considers curiosity signals from both the actor and the critic. We introduce the detailed formulations of actor and critic curiosity in Section 3.2 and Section 3.3 , respectively.

### 3.1 Challenge of Count-based exploration for RLVR

The core idea of count-based exploration is to measure the occurrence of Chain-of-Thought (CoT) patterns via sentence embeddings, and to assign an exploration bonus to rarely occurred CoTs. While conceptually appealing, this approach faces two major challenges:

• Curse of dimensionality: Classical count-based methods (e.g., LSVI-UCB ( Jin et al., 2020 ) , CFPO ( Cassel & Rosenberg, 2024 ) ) rely on computing the inverse of the covariance matrix Λ t − 1 \Lambda_{t}^{-1} (see Appendix E ) to construct historical visitation ellipsoids. For high-dimensional embeddings, this operation is computationally prohibitive.

To circumvent the need for matrix inversion, we investigated hash-based counts ( Tang et al., 2017 ) (details in Appendix C ), which project sentence embeddings into discrete hash grids and treat grid visitation frequency as a proxy for counts. However, this alternative introduces a second limitation:

• Poor expressiveness of embeddings: As illustrated in Figure 1 , after hash coding, most CoT embeddings collapse into neighboring hash grids. This clustering highlights the limited ability of sentence embeddings to distinguish between diverse reasoning patterns, leading to ineffective exploration.

In this work, we move beyond the paradigm of explicit state–action counts and instead utilize the model’s own measure of novelty. Our approach is motivated by the key intuition that agents, much like children in early cognitive development ( Chu & Schulz, 2020 ) , exhibit a form of curiosity . They respond with confidence when revisiting familiar states or CoT patterns, yet display uncertainty and exploratory behavior when confronted with novel situations. This learning process is not driven by a external count of experiences but is instead propelled by an intrinsic drive to explore. We first detail the implementation of the actor’s curiosity before turning to the critic’s.

### 3.2 Exploration Guided by Actor Curiosity

We model actor curiosity as the actor’s uncertainty about its own actions. Intuitively, a response that is surprising to the actor—i.e., has a low probability under its current policy—likely resides in an underexplored region of its learned distribution.

A natural and computationally efficient measure of this surprise is the perplexity of the actor’s generation. We formalize this as a sentence-level curiosity bonus, defined as the negative average log-probability of a generated sentence o = { o 1 , … , o T } o=\{o_{1},\ldots,o_{T}\} , given a prompt q q : B a ​ c ​ t ​ o ​ r ( q , o ) = − 1 T ∑ t = 1 T log π ( o t | o < t , q ) B_{actor}(q,o)=-\frac{1}{T}\sum_{t=1}^{T}\log\pi(o_{t}|o_{<t},q) (1) where π \pi denotes the actor policy. A higher value for B a ​ c ​ t ​ o ​ r ​ ( q , o ) B_{actor}(q,o) indicates greater surprise and thus a stronger intrinsic reward signal for exploration.