CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models — part 2/4

Cleaned-text code points [15014, 30991). Bibliography omitted here; source.md includes it.

However, practically simply adding this bonus to the original reward can be unstable and sub-optimal. Unconstrained exploration might incentivize the model to generate high-perplexity but low-quality or inaccurate responses (a behavior known as reward hacking), or lead to over-exploration where the policy fails to converge to a stable, high-quality output. To ensure that exploration remains tethered to the primary objective of maximizing the original reward signal, we integrate the bonus using an adaptive clipping mechanism. The total sentence-level reward, r ~ \tilde{r} , is a combination of the original reward signal r ⁡ ( q , o ) r(q,o) and the curiosity bonus B a ​ c ​ t ​ o ​ r ​ ( q , o ) B_{actor}(q,o) , where the bonus is capped relative to the original reward: r ~ ​ ( q , o ) = r ⁡ ( q , o ) + ω t ​ min ⁡ ( | r ⁡ ( q , o ) | κ , α ​ B a ​ c ​ t ​ o ​ r ​ ( q , o ) ) \tilde{r}(q,o)=r(q,o)+{\color[rgb]{1,0,0}\omega_{t}}\min(\frac{|r(q,o)|}{\color[rgb]{0,0,1}\kappa},{\color[rgb]{0,1,0}\alpha}B_{actor}(q,o)) (2)

This formulation promotes exploration by rewarding sentences that the actor finds surprising, while constraining the bonus to remain a fraction of the original reward. In this way, the model is discouraged from trading response quality for novelty. The behavior of this reward function is controlled by three key hyperparameters:

• The bonus weight ω t \omega_{t} is a dynamic coefficient, typically set with an annealing schedule to decrease over the course of training. This allows for more aggressive exploration in the early stages and then gradually shifts focus towards exploitation of high-reward regions as the policy converges.

• The clipping ratio κ \kappa governs the maximum size of the curiosity bonus relative to the original reward. By capping the bonus at | r ⁡ ( q , o ) | / κ |r(q,o)|/\kappa , it ensures the bonus remains a supplement and prevents it from dominating the learning signal. This is particularly crucial when r ⁡ ( q , o ) r(q,o) is negative, as it guarantees the bonus cannot reverse the sign of the reward, maintaining the integrity of the penalty.

• The bonus scaling factor α \alpha normalizes the curiosity bonus B actor ​ ( q , o ) B_{\text{actor}}(q,o) before it is compared to the clipped reward. A higher α \alpha allows the curiosity bonus to reach the clipping threshold more easily, whereas a smaller α \alpha diminishes its potential impact.

#### Intuitions and Theoretical Foundation

While the above formulation specifies how the perplexity bonus is shaped and controlled through its hyperparameters, it is equally important to understand its qualitative effect on model behavior. To build this intuition, we analyze responses along two axes: correctness and actor perplexity. Among these four categories, two require particular attention:

1. Incorrect responses with low PPL indicate that the model is highly confident in its answer, yet the response is wrong. This reflects overfitting and should be penalized.

2. Correct responses with high PPL suggest that the model is less familiar with such answers, but they nevertheless turn out to be successful. This reflects effective exploration and should be encouraged.

As illustrated in Figure 2 , we find out that the PPL bonus intrinsically penalizes confident mistakes while encouraging novel correct responses. For correct responses, those novel responses (with higher PPL) receive a larger positive reward. For incorrect responses, those confident responses (with lower PPL) receive larger penalty as it receives smaller PPL bonus. The following theorem formalizes this intuition; its precise statement and proof are deferred to Appendix D .

###### Theorem 3.1 .

Let π t \pi_{t} denote the policy at training step t. With PPL bonus in Equation ( 1 ), the update to π t + 1 \pi_{t+1} calibrates the policy’s confidence as follows:

(i) Among correct responses, trajectories with higher perplexity receive a larger relative probability increase.

(ii) Among incorrect responses, trajectories with lower perplexity receive a larger relative probability decrease.

Previous analyses distinguish the PPL bonus from the entropy bonus, which is sample-agnostic at the token level. The entropy at any given step depends solely on the policy’s probability distribution and is independent of the token ultimately sampled. Because the calculation considers the entire next-token distribution π θ ​ ( v ∣ q , o < t ) \pi_{\theta}\left(v\mid q,o_{<t}\right) (Equation 3 ), the bonus ℋ t \mathcal{H}_{t} remains constant for any potential outcome (Figure 3 ). Therefore, even when the model makes a high-confidence error by sampling token 1, the entropy bonus fails to penalize that choice. ℋ t = − ∑ v ∈ 𝒱 π θ ( v ∣ q , o < t ) log π θ ( v ∣ q , o < t ) . \mathcal{H}_{t}=-\sum_{v\in\mathcal{V}}\pi_{\theta}\left(v\mid q,o_{<t}\right)\log\pi_{\theta}\left(v\mid q,o_{<t}\right). (3)

### 3.3 Exploration Guided by Critic Curiosity

In contrast to critic-free methods such as REINFORCE and GRPO, the critic (value function) in actor–critic frameworks provides a higher-level understanding of the prompt–response pair by estimating the expected reward-to-go. Since this estimate is learned directly from collected trajectories, its posterior distribution conditioned on the observed data naturally reflects the degree of coverage: regions with dense data yield concentrated (low-variance) posteriors, whereas sparsely sampled regions result in higher uncertainty. Posterior distributions are a well-established means of quantifying predictive uncertainty in deep learning models ( Gal & Ghahramani, 2016 ; Lakshminarayanan et al., 2017 ) . As shown in Figure 4 , the orange curve exhibits lower variance—evidence of better data coverage—whereas the other curve is more dispersed, reflecting greater uncertainty.

To approximate the posterior distribution of value estimates, we adopt the classical bootstrap method ( Davison & Hinkley, 1997 ) , widely used in statistics and increasingly recognized in the RL community as an effective tool for exploration ( Osband et al., 2016 ; Ciosek et al., 2019 ; Bai et al., 2021 ) . We implement this idea through a multi-head critic (upper-left subfigure in Figure 10 ), where K K critics { V ^ 1 , … , V ^ K } \{\widehat{V}_{1},\ldots,\widehat{V}_{K}\} share a common LLM backbone. Each head is trained on a resampled subset of the collected trajectories (bottom subfigure in Figure 10 ), thereby producing an empirical approximation to the posterior distribution.

We then use the standard deviation across the K K heads as a principled curiosity signal, guiding the policy toward regions of high disagreement where the value function remains uncertain and under-explored. In the following theorem, we establish a surprising yet intuitive result: under a Linear MDP assumption, the standard deviation of the bootstrap critics is a consistent estimator of the pseudo-count bonus.

###### Theorem 3.2 .

In linear MDPs, the standard deviation across multi-head critics can serve as a consistent estimator for the pseudo-count exploration bonus, ϕ n , h ⊤ ​ Λ n , h − 1 ​ ϕ n , h \sqrt{\phi_{n,h}^{\top}\Lambda_{n,h}^{-1}\phi_{n,h}} , as used in LSVI-UCB ( Jin et al., 2020 ) and CFPO ( Cassel & Rosenberg, 2024 ) , where ϕ n , h = ϕ ⁡ ( s n , h , a n , h ) \phi_{n,h}=\phi(s_{n,h},a_{n,h}) is the feature vector of a state-action pair and Λ n , h = ∑ i = 0 n ϕ i , h ​ ϕ i , h ⊤ + λ ​ I \Lambda_{n,h}=\sum_{i=0}^{n}\phi_{i,h}\phi_{i,h}^{\top}+\lambda I is the coverage matrix.

The rigorous formulation of the linear MDP assumptions and the proof of Theorem 3.2 are provided in Appendix E , while empirical results in Section 4.4 further support this finding. Building on this foundation, we now describe the training procedure of the multi-head PPO algorithm, which follows the standard stages of vanilla PPO: (i) generating trajectories with the actor, (ii) updating the actor, and (iii) updating the critic. The key distinction is that we incorporate the multi-head variance as an exploration bonus, encouraging the policy to visit under-explored regions. A visual illustration of these steps is shown in Figure 5 .

• Actor roll-out: Given a prompt q q , the actor generates a set of responses { o 1 , … , o n } \{o_{1},\ldots,o_{n}\} . Each response is denoted as o i = { o i , 1 , … , o i , | o i | } o_{i}=\{o_{i,1},\ldots,o_{i,|o_{i}|}\} . Correspondingly, we associate each response with a verifiable reward r i r_{i} . For clarity, we focus on the case of a single prompt q q .

• Actor update: In this step, the advantage is estimated as A ^ i , t = ∑ l = t | o i | ( γ ​ λ ) l − t ​ δ ^ i , l ⏟ ≈ A ~ i , t + ω t ​ min ⁡ ( | A ~ i , t | κ , α ​ B critic ​ ( q , o i , ≤ t + 1 ) ) . \widehat{A}_{i,t}=\underbrace{\sum_{l=t}^{|o_{i}|}(\gamma\lambda)^{l-t}\widehat{\delta}_{i,l}}_{\approx\tilde{A}_{i,t}}+{\color[rgb]{1,0,0}\omega_{t}}\min\left(\frac{|\tilde{A}_{i,t}|}{\color[rgb]{0,0,1}\kappa},{\color[rgb]{0,1,0}\alpha}B_{\text{critic}}(q,o_{i,\leq t+1})\right). (4) The advantage consists of two components. The first term, A ~ i , t \tilde{A}_{i,t} , largely follows the standard advantage estimation in PPO, except that we exploit bootstrap estimators by using an ensemble of value functions rather than a single point estimate: δ ^ i , l = r i , l + γ K ​ ∑ j = 1 K V ^ j ​ ( q , o i , ≤ l + 1 ) − 1 K ​ ∑ j = 1 K V ^ j ​ ( q , o i , ≤ l ) . \widehat{\delta}_{i,l}=r_{i,l}+\frac{\gamma}{K}\sum_{j=1}^{K}\widehat{V}_{j}(q,o_{i,\leq l+1})-\frac{1}{K}\sum_{j=1}^{K}\widehat{V}_{j}(q,o_{i,\leq l}). The second term of Equation 4 introduces the multi-head critic bonus ( B critic B_{\text{critic}} ), governed by the bonus weight ω t \omega_{t} , clipping ratio κ \kappa , and scaling factor α \alpha (see discussion following Equation ( 2 ) for interpretation). Specifically, B critic B_{\text{critic}} is defined as the standard deviation across the K K value heads, encouraging exploration by assigning higher bonus to actions leading to uncertain/less-visited regions: B critic ​ ( q , o i , ≤ t + 1 ) = std ​ ( { V ^ j ​ ( q , o i , ≤ t + 1 ) | 1 ≤ j ≤ K } ) . B_{\text{critic}}\left(q,o_{i,\leq t+1}\right)=\text{std}\left(\big\{\widehat{V}_{j}(q,o_{i,\leq t+1})\big|1\leq j\leq K\big\}\right). (5)

• Critic update: We use the collected roll-outs to update the critic. For notational convenience, let the dataset be 𝒟 = { ( q , o i , ≤ t , r i ) | i ∈ [ n ] , t ∈ [ | o i | ] } , \mathcal{D}=\{(q,o_{i,\leq t},r_{i})|i\in[n],t\in[|o_{i}|]\}, (6) consisting of (prompt, partial response, reward) triplets. For each critic head j j , we sample without replacement a subset 𝒟 j ⊂ 𝒟 \mathcal{D}_{j}\subset\mathcal{D} of size | 𝒟 j | = ζ ​ | 𝒟 | |\mathcal{D}_{j}|=\zeta|\mathcal{D}| , where the hyperparameter ζ ∈ ( 0 , 1 ] \zeta\in(0,1] controls the fraction of data assigned per head. Smaller ζ \zeta increases head diversity, while larger ζ \zeta improves sample efficiency. The multi-head critic is then updated with the following bootstrap loss: ℒ ϕ = 1 ζ ​ K ​ | 𝒟 | ​ ∑ j = 1 K ∑ ( q , o , r ) ∈ 𝒟 j ( V ^ j ​ ( q , o ) − r ) 2 . \mathcal{L}_{\phi}=\frac{1}{\zeta K|\mathcal{D}|}\sum_{j=1}^{K}\sum_{(q,o,r)\in\mathcal{D}_{j}}\left(\widehat{V}_{j}(q,o)-r\right)^{2}.

## 4 Experiments

### 4.1 Dataset and Model

In this paper, we adopt DAPO-17K ( Yu et al., 2025 ) for training and evaluate the performance of CDE on four challenging mathematical reasoning benchmarks: MATH ( Hendrycks et al., 2021 ) , AMC23 ( MAA, b ) , AIME24, and AIME25 ( MAA, a ) . These evaluations are designed to assess CDE’s effectiveness in comparison to standard PPO and GRPO algorithms. Due to computational resource constraints, we conduct training with a reduced setting. All experiments are implemented within the Verl framework using the Qwen3-4B-Base model ( Yang et al., 2025 ) . For fair comparison, all the models use the default prompt in DAPO-17K as shown in Appendix B and the implementation details in Appendix A to further elaborate on the training settings.

### 4.2 Main Results

The main results are presented in Table 1 while the training dynamic is presented in Figure 6 . Here PPL bonus denote adding Curiosity bonus on actors as in Equation 2 , K K Heads represents multi-head critic PPO with K K head critics. We report both average Pass@1 accuracy and Pass@16 results on evaluation datasets. The key observations are as follows:

• The PPL bonus further enhances the mathematical reasoning ability of the GRPO method, yielding an average improvement of approximately + 2.4 +2.4 points across datasets and demonstrating consistent superiority. In particular, our method achieves notable gains on Pass@16, surpassing the baseline GRPO by about + 8 +8 points on the AIME24 dataset.

• Across benchmarks, multi-head PPO consistently outperforms vanilla PPO. Using K = 4 K=4 and K = 16 K=16 heads yields average gains of roughly + 2 +2 points, and we observe an around + 10 +10 points of increase in Pass@16 on AIME datasets in many cases.

• The performance of multi-head PPO generally increases with the number of heads K K : with K = 2 K=2 delivers negligible gains over the baseline, and performance increase begin to plateau once K ≥ 4 K\geq 4 , which suggests that a modest number of heads already captures most of the epistemic uncertainty needed.

• As shown in Figure 6 , GRPO with PPL bonus and multi-head PPO increase test accuracy more slowly than baseline PPO/GRPO early in training, then catch up and ultimately surpass them. This pattern is consistent with enhanced exploration: the PPL bonus and head disagreement discourage premature exploitation of spurious high-reward trajectories. As state-action coverage expands, these signals calibrate, enabling a smoother shift to targeted exploitation and yielding higher final accuracy.

### 4.3 Understanding the Effect of the PPL Bonus

In this subsection, we present additional experiments to investigate the role of the PPL bonus, from which we derive the following key findings.

Bonus weight decay is crucial We compare four schedules for the bonus weight ω t \omega_{t} — No decay , Linear , Cosine , and Staircase —as illustrated in Figure 7 , with the performance of models trained under each schedule summarized in Table 2 . Briefly, the No decay schedule maintains strong exploration throughout training, while the Staircase schedule reduces ω t \omega_{t} abruptly, enabling strong exploration in the early phase and then removing the bonus for final convergence. The Linear and Cosine schedules provide intermediate behaviors.

The results in Table 2 underscore two insights: First, decay of the bonus weight is necessary, as all decay schedules outperform the no-decay baseline by enabling a gradual shift from exploration to exploitation. Second, strong exploration in the early phase is crucial, with the staircase scheme proving most effective by sustaining high exploration initially to broaden state–action coverage and then removing the bonus abruptly to allow stable convergence, whereas the gentler cosine and linear decays weaken the signal too soon and thus yield smaller gains.

Analysis of Entropy Dynamics As highlighted in prior work, entropy provides an important lens for understanding exploration ability ( Cui et al., 2025 ) , where a sharp decline in entropy often signals premature convergence and insufficient exploration. Figure 8 illustrates the entropy dynamics of baseline GRPO compared to our proposed methods. First, relative to the baseline, the PPL bonus alleviates entropy collapse, demonstrating its role in promoting exploration. Second, when comparing decay schemes, PPL with No Decay shows persistent fluctuations and fails to converge, whereas Staircase decay yields more stable entropy trajectories. This observation is consistent with our earlier findings that decaying the bonus weight is essential for ensuring stable convergence while still supporting effective exploration.