CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models — part 3/4

Cleaned-text code points [30991, 46513). Bibliography omitted here; source.md includes it.

Analysis of Calibration As shown in Figure 9 , we plot the batch-wise mean response perplexity (PPL), stratified by answer correctness. In subfigure (a), we observe a phenomenon we term calibration collapse : early in naive GRPO training, correct responses have lower PPL (higher confidence) than incorrect ones, but as training progresses this gap shrinks and ultimately vanishes—confidence no longer tracks correctness. By contrast, with a PPL bonus (subfigure (b)), this separation is sustained throughout training.

This pattern is explained by Theorem 3.1 : while both naive GRPO and GRPO with a PPL bonus tend to increase confidence on correct answers, the PPL bonus additionally suppresses confident errors (low-PPL incorrect trajectories), thereby improving calibration.

This finding is original and practically important. Ideally, a trained model should be faithful—confident when its answer is correct and cautious when it is not. Better calibration enhances interpretability and supports inference-time selection strategies such as self-certainty BoN ( Wang et al., 2022 ) and DeepConf ( Fu et al., 2025 ) . It also connects to the growing literature on calibrating LLMs, both during training (e.g., ( Shen et al., 2024 ) ) and at test time (e.g., ( Ulmer et al., 2024 ) ).

### 4.4 Further Analysis of the Multi-Head Critic

Analysis of Dynamics of B critic B_{\text{critic}} We further examine the dynamics of the multi-head exploration bonus B critic B_{\text{critic}} by tracking its average value over the course of training. Specifically, for each training step, given the roll-outs 𝒟 \mathcal{D} defined in Equation 6 , we compute the average B critic B_{\text{critic}} across (prompt, partial response, reward) triplets within 𝒟 \mathcal{D} . As shown in sub-figure (a) of Figure 10 , this average decreases steadily as training progresses. The decline reflects that, with more training, similar trajectories are revisited more frequently, leading to reduced disagreement among critic heads. This phenomenon provides empirical support for interpreting the multi-head bonus as analogous to count-based exploration measures.

In sub-figure (b) of Figure 10 , we present a cross-dataset analysis by calculating the average standard deviation of the value estimates across different questions. Specifically, we evaluate three datasets: the training set (DAPO-17K), the in-domain validation set (AMC23), and the out-of-domain validation set GPQA ( Rein et al., 2023 ) . We observe that the training set exhibits a smaller standard deviation compared to both the in-domain and out-of-domain validation sets. This pattern aligns with the intuition that multi-head critics tend to show stronger disagreement on data that is less frequently encountered during training.

Analysis of sub-sample fraction ζ \zeta during critic update Additionally, we examine the sensitivity of the critic update to the hyperparameter ζ \zeta (sub-sample fraction). We vary ζ \zeta under two configurations—critics with 16 heads and with 4 heads—and compare ζ ∈ { 0.5 , 1 } \zeta\in\{0.5,1\} . As shown in Table 3 , while a larger number of heads benefits from a larger sub-sample fraction, the overall performance is stable across settings. The model demonstrates robustness to the masking fraction ζ \zeta , achieving similar results for both values tested (0.5 and 1.0).

## 5 Related Work

### 5.1 Reinforcement Learning (RL) for LLM reasoning

Reinforcement Learning is a central technique for advancing the reasoning capabilities of LLMs. Initial approaches relied on reward models that provided either outcome-based supervision, focusing on the final answer ( Cobbe et al., 2021 ) , or process-based supervision, evaluating intermediate reasoning steps ( Uesato et al., 2022 ) . To navigate more complex problem spaces, these foundational reward strategies were often augmented with search algorithms such as MCTS ( Feng et al., 2023 ; Tian et al., 2024 ; Chen et al., 2024 ; Wang et al., 2024c ) and Q* ( Wang et al., 2024b ; Wang et al., 2024a ) . More recently, RLVR ( Lambert et al., 2024 ) has emerged as a powerful alternative, demonstrating significant performance on complex reasoning tasks in mathematics and coding ( Guo et al., 2025 ) . Consequently, a growing body of work seeks to apply RLVR to diverse domains, including multi-modal reasoning ( Wang et al., 2025 ; Li et al., 2025 ) , logical reasoning ( Zhou et al., 2025 ) , search engine use ( Jin et al., 2025 ; Xiong et al., 2025 ) , and information extraction ( Dai et al., 2025b ) . Parallel efforts aim to improve upon the standard RLVR paradigm with techniques such as mixture-of-thought ( Zheng et al., 2025a ) , self-evolving ( Huang et al., 2025 ) , parallel thinking ( Zheng et al., 2025b ) . Despite these advances, persistent concerns remain regarding robustness ( Dai et al., 2025a ; Zhao et al., 2025 ) , calibration ( Shen et al., 2024 ) , and a lack of exploration evidenced by entropy collapse ( Cui et al., 2025 ; Shen, 2025 ) , highlighting the need for more principled training approaches.

### 5.2 Efficient exploration

Efficient exploration is a central challenge in Reinforcement Learning (RL), which aim to balance between exploration and exploitation ( Sutton & Barto, 2018 ; Weng, 2020 ; Amin et al., 2021 ) . Many foundational approaches are heuristic-based, such as Gaussian noise ( Lillicrap et al., 2015 ) or the ϵ \epsilon -greedy method ( Sutton & Barto, 2018 ) . Entropy regularization is a more principled heuristic, which encourages the policy to be more stochastic. While simple to implement, these methods are often undirected—they promote pure randomness. Consequently, they can be suboptimal ( Dann et al., 2022 ) with no significant gains in complex Deep RL ( Andrychowicz et al., 2021 ) or LLM training ( Cui et al., 2025 ; Shen, 2025 ) .

In contrast, a major class of methods incentivizes exploration by adding exploration bonus to guide the agent toward novel or uncertain parts of the environment. Count-based approaches like UCB ( Lai, 1987 ) , LinUCB ( Li et al., 2010 ) , and LSVI-UCB ( Jin et al., 2020 ) use pseudo-counts of state-action visitations to encourage exploring rarely visited areas, achieving near-optimal theoretical guarantees in bandits and linear MDPs. Similarly, prediction-based methods such as ICM ( Pathak et al., 2017 ) and RND ( Burda et al., 2018 ) use the error from a predictive model as a bonus, rewarding the agent for reaching states that are difficult to predict. Applying these guided exploration principles is a growing field in LLM. For instance, Bai et al. (2025) incorporate a count-based bonus into the RLHF process by introducing a coin flipping module. Gao et al. (2025) draws inspiration from RND by adding an auxiliary noise prediction network. However, both methods rely on expressive representations of long COT trajectories and introduce additional modules, which complicates the training framework. In contrast, CDE uses intrinsic curiosity signals from the actor and critics, requiring only minimal modifications to the framework and yielding efficient exploration both theoretically and empirically.

## 6 Conclusion and Future Work

We have presented Curiosity-Driven Exploration, an efficient technique that enhances agent learning by incorporating curiosity signals from both the actor and the critic. Our approach is notably lightweight, demanding only minor modifications to the original training architecture. Its effectiveness is demonstrated by consistent accuracy improvements over strong baselines on a suite of challenging mathematical reasoning benchmarks, with these empirical results strongly corroborating our underlying theoretical framework and intuition.

The calibration collapse revealed in our analysis aligns with recent findings on the root causes of LLM hallucination ( Kalai et al., 2025 ) , pointing to a promising avenue for future work. We hypothesize that the underlying source of this collapse is the reward design of RLVR training. Specifically, RLVR with outcome reward prioritizes correct final outcomes at the expense of rigorous intermediate reasoning. Our experiments shed light on this direction by demonstrating that an alternative multi-perspective reward design (e.g., the PPL bonus) can be valuable for guiding the RLVR process more effectively.

## Appendix A Training Details

We use verl as the training framework 1 1 1 https://github.com/volcengine/verl . Configurations for training CDE and baseline models are listed in Table 4 .

## Appendix B Prompt

Figure 11: The prompt for RLVR training.

## Appendix C Details on Hash-based pseudo count

The core idea is to map a full prompt–response trajectory to a compact hash that serves as a pseudo-state for exploration. Given a prompt–response pair ( q , o ) (q,o) with tokenized sequences q = { q 1 , … , q D } q=\{q_{1},\ldots,q_{D}\} and o = { o 1 , … , o T } o=\{o_{1},\ldots,o_{T}\} , let the model produce last-layer hidden states 𝕙 = { h 1 , … , h D + T } \mathbb{h}=\{h_{1},\ldots,h_{D+T}\} , h i ∈ ℝ d h_{i}\in\mathbb{R}^{d} . We form a trajectory embedding h q , o ∈ ℝ d h_{q,o}\in\mathbb{R}^{d} from 𝕙 \mathbb{h} via one of: (i) h D + T h_{D+T} ; (ii) h D + T − 1 h_{D+T-1} ; or (iii) mean pooling 1 D + T ​ ∑ i = 1 D + T h i \frac{1}{D+T}\sum_{i=1}^{D+T}h_{i} . With a random projection matrix A ∈ ℝ k × d A\in\mathbb{R}^{k\times d} (rows drawn i.i.d. from 𝒩 ⁡ ( 0 , I ) \mathcal{N}(0,I) or Rademacher), we compute a k k -bit SimHash code ϕ ⁡ ( q , o ) = sign ⁡ ( A ​ h q , o ) ∈ { − 1 , + 1 } k , \phi(q,o)=\operatorname{sign}\!\big(Ah_{q,o}\big)\in\{-1,+1\}^{k}, and map it to a bucket index b = bucket ⁡ ( ϕ ) ∈ { 0 , … , 2 k − 1 } b=\mathrm{bucket}(\phi)\in\{0,\ldots,2^{k}-1\} . Let n ⁡ ( b ) n(b) be the visitation count of bucket b b . We apply intrinsic reward shaping to encourage rarely visited trajectories: r ~ ​ ( q , o ) = r ⁡ ( q , o ) + ω t ​ β t n ⁡ ( b ) , \tilde{r}(q,o)=r(q,o)+\omega_{t}\frac{\beta_{t}}{\sqrt{n(b)}}, where β t \beta_{t} is the weight for exploration bonus a. This yields an efficient, matrix-inversion–free intrinsic bonus that scales linearly in k ​ d kd per sample, following Tang et al. (2017) .

## Appendix D Proof for Calibration Theorem

Define r ~ t ​ ( q , o ) = r ⁡ ( q , o ) + b t ​ ( q , o ) \tilde{r}_{t}(q,o)=r(q,o)+b_{t}(q,o) where b t ​ ( q , o ) = ω ​ min ⁡ { κ ​ | r ⁡ ( q , o ) | , − α T o ​ log ⁡ π t ​ ( o | q ) } b_{t}(q,o)=\omega\min\{\kappa|r(q,o)|,-\frac{\alpha}{T_{o}}\log\pi_{t}(o|q)\} is a bonus function where T o T_{o} is the length of response o o . Note that ω \omega is a redundant variable in theory because we can write b t ​ ( q , o ) = min ⁡ { κ ′ ​ | r ⁡ ( q , o ) | , − α T o ​ log ⁡ π t ​ ( o | q ) } b_{t}(q,o)=\min\{\kappa^{\prime}|r(q,o)|,-\frac{\alpha}{T_{o}}\log\pi_{t}(o|q)\} with κ ′ = ω ​ κ \kappa^{\prime}=\omega\kappa and α ′ = ω ​ α \alpha^{\prime}=\omega\alpha . Given that r ⁡ ( x , y ) ∈ { 1 , − 1 } r(x,y)\in\{1,-1\} , it suffices to consider b t ​ ( q , o ) = min ⁡ { κ , − α T o ​ log ⁡ π t ​ ( o | q ) } b_{t}(q,o)=\min\{\kappa,-\frac{\alpha}{T_{o}}\log\pi_{t}(o|q)\} . Thus, as long as we use κ < 1 \kappa<1 , we have sign ​ ( r ~ t ​ ( q , o ) ) = sign ​ ( r ⁡ ( q , o ) ) \text{sign}(\tilde{r}_{t}(q,o))=\text{sign}(r(q,o)) . The introduce of bonus does not change the sign of the original correctness reward.

Consider single step policy optimization π t + 1 ( ⋅ | q ) = arg max π { ∑ o π ( o | q ) r ~ t ( q , o ) − 1 η KL ( π ( ⋅ | q ) ∥ π t ( ⋅ | q ) ) } , \displaystyle\pi_{t+1}(\cdot|q)=\arg\max_{\pi}\left\{\sum_{o}\pi(o|q)\tilde{r}_{t}(q,o)-\frac{1}{\eta}\text{KL}\left(\pi(\cdot|q)\|\pi_{t}(\cdot|q)\right)\right\}, which has closed-form solution π t + 1 ​ ( o | q ) = π t ​ ( o | q ) ​ exp ⁡ ( η ​ r ~ t ​ ( q , o ) ) ∑ o ′ π t ​ ( o ′ | q ) ​ exp ⁡ ( η ​ r ~ t ​ ( q , o ′ ) ) . \displaystyle\pi_{t+1}(o|q)=\frac{\pi_{t}(o|q)\exp\left(\eta\tilde{r}_{t}(q,o)\right)}{\sum_{o^{\prime}}\pi_{t}(o^{\prime}|q)\exp\left(\eta\tilde{r}_{t}(q,o^{\prime})\right)}. For any question q q and response o o . Define Z ⁡ ( q ) = ∑ o ′ π t ​ ( o ′ | q ) ​ exp ⁡ ( η ​ r ~ t ​ ( q , o ′ ) ) Z(q)=\sum_{o^{\prime}}\pi_{t}(o^{\prime}|q)\exp\left(\eta\tilde{r}_{t}(q,o^{\prime})\right) , we have log ⁡ π t + 1 ​ ( o | q ) = log ⁡ π t ​ ( o | q ) + η ​ r ~ t ​ ( q , o ) − log ⁡ ( Z ⁡ ( q ) ) . \displaystyle\log\pi_{t+1}(o|q)=\log\pi_{t}(o|q)+\eta\tilde{r}_{t}(q,o)-\log\left(Z(q)\right). Define Δ t ​ ( o | q ) = log ⁡ π t + 1 ​ ( o | q ) − log ⁡ π t ​ ( o | q ) \Delta_{t}(o|q)=\log\pi_{t+1}(o|q)-\log\pi_{t}(o|q) as the change of likelihood of response o o under question q q at update step t t . For two correct response o 1 + o^{+}_{1} and o 2 + o^{+}_{2} with length T o 1 + T_{o^{+}_{1}} and T o 2 + T_{o^{+}_{2}} , and − α T o 1 + ​ log ⁡ π t ​ ( o 1 + | q ) ≥ − α T o 2 + ​ log ⁡ π t ​ ( o 2 + | q ) -\frac{\alpha}{T_{o^{+}_{1}}}\log\pi_{t}(o^{+}_{1}|q)\geq-\frac{\alpha}{T_{o^{+}_{2}}}\log\pi_{t}(o^{+}_{2}|q) (i.e. o 1 + o_{1}^{+} has larger perplexity), we have Δ t ​ ( o 1 + | q ) − Δ t ​ ( o 2 + | q ) \displaystyle\Delta_{t}(o^{+}_{1}|q)-\Delta_{t}(o^{+}_{2}|q) = r ~ t ​ ( q , o 1 + ) − r ~ t ​ ( q , o 2 + ) \displaystyle=\tilde{r}_{t}(q,o^{+}_{1})-\tilde{r}_{t}(q,o^{+}_{2}) = b t ​ ( q , o 1 + ) − b t ​ ( q , o 2 + ) \displaystyle=b_{t}(q,o^{+}_{1})-b_{t}(q,o^{+}_{2}) = min ⁡ { κ , − α T o 1 + ​ log ⁡ π t ​ ( o 1 + | q ) } − min ⁡ { κ , − α T o 2 + ​ log ⁡ π t ​ ( o 2 + | q ) } \displaystyle=\min\{\kappa,-\frac{\alpha}{T_{o^{+}_{1}}}\log\pi_{t}(o^{+}_{1}|q)\}-\min\{\kappa,-\frac{\alpha}{T_{o^{+}_{2}}}\log\pi_{t}(o^{+}_{2}|q)\} ≥ 0 \displaystyle\geq 0

Similarly, for two incorrect response o 1 − o^{-}_{1} and o 2 − o^{-}_{2} with − α T o 1 − ​ log ⁡ π t ​ ( o 1 − | q ) ≥ − α T o 2 − ​ log ⁡ π t ​ ( o 2 − | q ) -\frac{\alpha}{T_{o^{-}_{1}}}\log\pi_{t}(o^{-}_{1}|q)\geq-\frac{\alpha}{T_{o^{-}_{2}}}\log\pi_{t}(o^{-}_{2}|q) (i.e. o 1 − o_{1}^{-} has larger perplexity), we have Δ t ​ ( o 1 − | q ) − Δ t ​ ( o 2 − | q ) ≥ 0 \Delta_{t}(o^{-}_{1}|q)-\Delta_{t}(o^{-}_{2}|q)\geq 0 .

Specifically, given a question q q , for any response ( o 1 , o 2 ) (o_{1},o_{2}) that has the same correctness label and − α T o 1 ​ log ⁡ π t ​ ( o 1 | q ) ≥ − α T o 2 ​ log ⁡ π t ​ ( o 2 | q ) -\frac{\alpha}{T_{o_{1}}}\log\pi_{t}(o_{1}|q)\geq-\frac{\alpha}{T_{o_{2}}}\log\pi_{t}(o_{2}|q) , we have • If r ~ t ​ ( q , o 1 ) ≥ 1 η ​ log ⁡ ( Z ⁡ ( q ) ) \tilde{r}_{t}(q,o_{1})\geq\frac{1}{\eta}\log\left(Z(q)\right) and r ~ t ​ ( q , o 2 ) ≥ 1 η ​ log ⁡ ( Z ⁡ ( q ) ) \tilde{r}_{t}(q,o_{2})\geq\frac{1}{\eta}\log\left(Z(q)\right) , then Δ t ​ ( o 1 | q ) ≥ 0 \Delta_{t}(o_{1}|q)\geq 0 and Δ t ​ ( o 2 | q ) ≥ 0 \Delta_{t}(o_{2}|q)\geq 0 but o 1 o_{1} has more likelihood increase.

• If r ~ t ​ ( q , o 1 ) ≥ 1 η ​ log ⁡ ( Z ⁡ ( q ) ) \tilde{r}_{t}(q,o_{1})\geq\frac{1}{\eta}\log\left(Z(q)\right) and r ~ t ​ ( q , o 2 ) < 1 η ​ log ⁡ ( Z ⁡ ( q ) ) \tilde{r}_{t}(q,o_{2})<\frac{1}{\eta}\log\left(Z(q)\right) , then Δ t ​ ( o 1 | q ) ≥ 0 \Delta_{t}(o_{1}|q)\geq 0 and Δ t ​ ( o 2 | q ) < 0 \Delta_{t}(o_{2}|q)<0 where o 1 o_{1} ’s likelihood increase but o 2 o_{2} ’s likelihood decrease.

• If r ~ t ​ ( q , o 1 ) < 1 η ​ log ⁡ ( Z ⁡ ( q ) ) \tilde{r}_{t}(q,o_{1})<\frac{1}{\eta}\log\left(Z(q)\right) and r ~ t ​ ( q , o 2 ) < 1 η ​ log ⁡ ( Z ⁡ ( q ) ) \tilde{r}_{t}(q,o_{2})<\frac{1}{\eta}\log\left(Z(q)\right) , then Δ t ​ ( o 1 | q ) < 0 \Delta_{t}(o_{1}|q)<0 and Δ t ​ ( o 1 | q ) < 0 \Delta_{t}(o_{1}|q)<0 but o 1 o_{1} has less likelihood decrease.