Reinforced Efficient Reasoning via Semantically Diverse Exploration β€” part 2/3

Cleaned-text code points [15523, 31436). Bibliography omitted here; source.md includes it.

Node value assignment. After completing tree-structured sampling, a single response may contain multiple branching nodes. Including the start and the terminal positions, these nodes partition a response into several consecutive segments. Formally, for a response 𝐨 i \mathbf{o}_{i} to a given question q q , let b 0 , b 1 , … , b k {b_{0},b_{1},\dots,b_{k}} denote the node positions, where b 0 b_{0} is the start position, b 1 , … , b k βˆ’ 1 b_{1},\dots,b_{k-1} correspond to the pivot positions immediately before each branching point, and b k b_{k} is the leaf (terminal) position. The response can then be decomposed as 𝐨 i = ⋃ j = 1 k 𝐨 i , b j βˆ’ 1 < t ≀ b j , with ​ b 0 = 0 , b k = | 𝐨 i | \mathbf{o}_{i}=\bigcup_{j=1}^{k}\mathbf{o}_{i,b_{j-1}<t\leq b_{j}},\text{with }b_{0}=0,b_{k}=|\mathbf{o}_{i}| (4) Under this partition, each segment is initiated at either the start position or a branching position selected based on maximal semantic entropy observed during the rollout stage. For a pivot node b j b_{j} with 0 < j < k 0<j<k , we define the set of responses that contain this node as: Ξ© b j = { 𝐨 m | 𝐨 m ​ traverses the pivot node ​ b j } \Omega_{b_{j}}=\{\mathbf{o}_{m}|\mathbf{o}_{m}\text{ traverses the pivot node }b_{j}\} (5) For the start node b 0 b_{0} , we define Ξ© b 0 \Omega_{b_{0}} as the set of all responses, i.e., Ξ© b 0 = { 𝐨 m } m = 1 G \Omega_{b_{0}}=\{\mathbf{o}_{m}\}_{m=1}^{G} . For the leaf node, we define Ξ© b k \Omega_{b_{k}} as the set of 𝐨 i \mathbf{o}_{i} , i.e., Ξ© b k = { 𝐨 i } \Omega_{b_{k}}=\{\mathbf{o}_{i}\} .

Next, we define the value of node b j b_{j} with 0 ≀ j ≀ k 0\leq j\leq k as the average reward of responses in Ξ© b j \Omega_{b_{j}} . V ^ ​ ( b j ) = 1 | Ξ© b j | ​ βˆ‘ 𝐨 m ∈ Ξ© b j r ⁑ ( 𝐨 m ) \hat{V}(b_{j})=\frac{1}{|\Omega_{b_{j}}|}\sum_{\mathbf{o}_{m}\in\Omega_{b_{j}}}r(\mathbf{o}_{m}) (6) where r ⁑ ( ) r() denotes a rule-based reward function that evaluates the correctness of each response and assigns a binary reward (1 for correct and 0 for incorrect).

Segment advantage estimation. Next, we can compute the segment-level advantage between two nodes based on the values assigned to the nodes. According to the definition of node value, the value of a node can be interpreted as the probability of deriving a correct reasoning chain starting from that node. Therefore, the reasoning contribution of the segment is quantified by the difference between the two node values. Specifically, for any token 𝐨 i , t ∈ 𝐨 i \mathbf{o}_{i,t}\in\mathbf{o}_{i} with b j βˆ’ 1 < t ≀ b j b_{j-1}<t\leq b_{j} , the advantage of 𝐨 i , t \mathbf{o}_{i,t} is defined as: A ^ i , t = V ^ ​ ( b j ) βˆ’ V ^ ​ ( b j βˆ’ 1 ) \hat{A}_{i,t}=\hat{V}(b_{j})-\hat{V}(b_{j-1}) (7)

Length-aware calibration. Furthermore, although multiple reasoning paths may lead to correct outcomes, we aim to encourage the model to adopt more efficient reasoning and avoid overthinking. To this end, we apply a length-aware calibration to the advantages of responses that are correct but require an excessive number of tokens. Specifically, we first identify the shortest correct response 𝐨 s \mathbf{o}_{s} . Then, for every other correct response 𝐨 c \mathbf{o}_{c} , we locate the pivot node b c b_{c} at which 𝐨 s \mathbf{o}_{s} and 𝐨 c \mathbf{o}_{c} diverge. That is, prior to b c b_{c} , the two responses share an identical subsequence 𝐨 s , ≀ b c = 𝐨 c , ≀ b c \mathbf{o}_{s,\leq b_{c}}=\mathbf{o}_{c,\leq b_{c}} , whereas after b c b_{c} they follow distinct continuations 𝐨 s , > b c \mathbf{o}_{s,>b_{c}} and 𝐨 c , > b c \mathbf{o}_{c,>b_{c}} , respectively. A length-proportional calibration is then applied to the longer response, thereby encouraging the model to produce more efficient reasoning. Specifically, for each token 𝐨 c , t ∈ 𝐨 c \mathbf{o}_{c,t}\in\mathbf{o}_{c} with t > b c t>b_{c} , its advantage is updated according to the following rule: A ^ i , t ← A ^ i , t βˆ’ | A ^ i , t | β‹… ( 1 βˆ’ ( | 𝐨 s | βˆ’ b c | 𝐨 c | βˆ’ b c ) Ξ± ) \hat{A}_{i,t}\leftarrow\hat{A}_{i,t}-|\hat{A}_{i,t}|\cdot(1-(\frac{|\mathbf{o}_{s}|-b_{c}}{|\mathbf{o}_{c}|-b_{c}})^{\alpha}) (8) where Ξ± \alpha is a hyperparameter controlling the extent of the adjustment. The ratio | 𝐨 s | βˆ’ b c | 𝐨 c | βˆ’ b c \frac{|\mathbf{o}_{s}|-b_{c}}{|\mathbf{o}_{c}|-b_{c}} measures the relative lengths of the two reasoning branches after their divergence at b c b_{c} . Since the two responses share an identical reasoning prefix before b c b_{c} , their post-divergence segments can be directly compared: the more efficient branch receives a higher advantage, while the longer branch incurs a length-proportional adjustment.

### 3.3 Model Training

We adopt the improved modifications of vanilla GRPO’s optimizing objective proposed in Dr.GRPO Liu et al. (2025) as the training objective, together with a KL penalty term:

β„’ ROSE ​ ( ΞΈ ) = βˆ’ 1 G βˆ‘ i = 1 G βˆ‘ t = 1 | 𝐨 i | ( min ( r i , t ( ΞΈ ) A ^ i , t , [ r i , t ( ΞΈ ) ] 1 βˆ’ Ο΅ β€² 1 + Ο΅ β€² A ^ i , t ) βˆ’ Ξ² D KL ( Ο€ ΞΈ βˆ₯ Ο€ ref ) ) , \begin{aligned} \mathcal{L}_{\text{ROSE }}(\theta)&=-\frac{1}{G}\sum_{i=1}^{G}\sum_{t=1}^{|\mathbf{o}_{i}|}\Bigg(\min\Big(r_{i,t}(\theta)\hat{A}_{i,t},[r_{i,t}(\theta)]_{1-\epsilon^{\prime}}^{1+\epsilon^{\prime}}\hat{A}_{i,t}\Big)\\ \hskip 0.0pt&-\beta D_{\text{KL}}(\pi_{\theta}\|\pi_{\text{ref}})\Bigg),\end{aligned}

where

Ο€ old \pi_{\text{old}} denotes sampling model, Ο€ ref \pi_{\text{ref}} denotes reference model and operator [ r i , t ​ ( ΞΈ ) ] 1 βˆ’ Ο΅ β€² 1 + Ο΅ β€² [r_{i,t}(\theta)]_{1-\epsilon^{\prime}}^{1+\epsilon^{\prime}} clips the ratio to [ 1 βˆ’ Ο΅ β€² , 1 + Ο΅ β€² ] [1-\epsilon^{\prime},1+\epsilon^{\prime}] .

## 4 Experimental Setup

### 4.1 Datasets and Metrics

For the training dataset, following prior studies Zhu et al. (2025) ; Liu et al. (2025) , we use MATH Hendrycks et al. (2021) , which contains 7,500 problems. For evaluation, four publicly available standard mathematical reasoning benchmarks are considered, including AIME2024, AIME2025, AMC23, and MATH500. MATH500 is a subset of the test split of the MATH dataset, consisting of 500 problems. During validation, we sample 8 responses for each question and adopt pass@8 as the primary metric for assessing the performance of model reasoning. pass@ k k measures whether at least one of the k k sampled responses correctly solves a given problem. Unlike prior work that relies on the mean@ k k metric Li et al. (2025) ; Yang et al. (2025b) , which reflects the average accuracy across all samples, pass@ k k more effectively captures the model’s ability to solve previously challenging problems that it might fail to answer.

### 4.2 Model and Baselines

To provide a more comprehensive comparison of the proposed method, we evaluate it using backbone models from two model families, Qwen and Llama, with different parameter scales, including Llama-3.2-3B-Instruct Grattafiori et al. (2024) , Qwen3-4B-Base , and Qwen3-8B-Base Yang et al. (2025a) . The Qwen3 models have two modes (thinking and non-thinking), and the non-thinking mode is adopted for both training and inference.

Comparisons are performed between ROSE and existing approaches, which mainly fall into two categories: GRPO-based variants and MCTS-based methods. The GRPO-based variants include vanilla GRPO Guo et al. (2025) ; Shao et al. (2024) , Dr.GRPO Liu et al. (2025) , and DAPO Yu et al. (2025) . Dr.GRPO computes advantages as deviations from the group mean, without variance normalization, and removes the length normalization term from the loss function. DAPO improves GRPO by incorporating techniques such as clip-higher and rejection sampling.

The MCTS-based baselines include FR3E Zheng et al. (2025b) and TreePO Li et al. (2025) . FR3E is a representative MCTS-based method that determines branching positions based on generation entropy and adopts a two-step framework for segment-level advantage computation. Besides, TreePO structures the rollout process as a tree by branching at fixed-length segments and computes advantages over the resulting sub-trees.

### 4.3 Implementation Details

All experiments are conducted using the VeRL framework Sheng et al. (2025) in this paper. For RL training, we set the batch size to 512, the number of rollouts per prompt as G = 8 G=8 , the learning rate to 1 Γ— 10 βˆ’ 6 1\times 10^{-6} , the clipping ratio as Ο΅ β€² = 0.2 \epsilon^{\prime}=0.2 , the KL divergence coefficient as Ξ² = 0.001 \beta=0.001 , and the maximum number of training epochs to 8. For evaluation, the temperature is set to 0.6, top- p p sampling is applied with p = 0.95 p=0.95 , and 8 candidate responses are sampled per prompt. Prompts whose lengths exceed 2048 tokens are filtered out, and the maximum generation length is set to 4096 tokens. The probability of generating the response from scratch Ξ΅ \varepsilon is set to 0.5 by default, and the coefficient for length-aware calibration Ξ± \alpha is searched from { 0.5 , 1 , 2 , 3 } \{0.5,1,2,3\} . Our experiments are conducted on 8 Γ— \times NVIDIA A800 (80G) GPUs.

## 5 Experimental Results

### 5.1 Overall Performance

Accuracy evaluation. Table 1 presents the experimental results of all methods. It can be observed that ROSE achieves substantial improvements over the strongest baseline in most settings. DAPO and Dr.GRPO are variants that modify the GRPO loss function. However, they do not yield consistent or substantial improvements, with performance gains observed only in certain scenarios, such as models with larger parameter scales. Among MCTS-based approaches, TreePO and FR3E achieve performance comparable to GRPO and its variants. In particular, TreePO yields pronounced improvements on the in-domain dataset MATH500 but performs worse on other benchmarks. This suggests that its fixed-length branching strategy fails to induce more diverse reasoning trajectories during exploration, limiting out-of-domain generalization.

For ROSE, significant performance gains are first observed on more challenging tasks, indicating that the method facilitates more divergent exploration during the rollout phase, which is beneficial for solving difficult problems. In addition, ROSE consistently yields notable improvements across different model scales. Larger models typically encapsulate richer knowledge, and the proposed approach appears to leverage this capacity more effectively, resulting in greater performance gains. Finally, recent studies have suggested that models in the Qwen family may suffer from potential data leakage Wu et al. (2025a) . Nevertheless, comparable performance gains are also observed on Llama models of similar parameter scales, indicating that the improvements are not confined to a specific model family.

Learning dynamics. Figure 3 presents the learning curves of GRPO and ROSE. Across different model scales, ROSE exhibits a clear performance improvement over the vanilla GRPO. Moreover, as the model scale increases, the learning curves of ROSE become noticeably more stable. After convergence, ROSE maintains stable and competitive performance, whereas the vanilla GRPO shows noticeable fluctuations and fails to achieve significant improvements in small-scale ( 3B and 4B ) models.

### 5.2 Branching Metric Analysis

Performance comparison. Table 2 presents the experimental results for three different branching metrics (entropy, semantic divergence, and semantic entropy). For all branching metrics, the probability Ξ΅ \varepsilon is fixed to 0.5. The results show that entropy achieves performance comparable to vanilla GRPO, indicating that it fails to effectively distinguish uncertain regions in the model’s reasoning trajectories. In contrast, both semantic divergence and semantic entropy yield consistent improvements over entropy, with semantic entropy exhibiting greater robustness across different datasets.

Analysis of reasoning diversity. To further analyze the differences among branching metrics, we conduct a quantitative analysis of the diversity of responses generated during the rollout phase under different branching metrics. Specifically, for a fixed batch of questions, rollouts are performed using different metrics, and the pairwise cosine similarity between embeddings of multiple responses corresponding to the same question is computed. The embeddings are obtained using the Qwen-text-embedding-v4 model. The resulting similarity distributions are visualized using kernel density estimation (KDE), as shown in Figure 4 .

As illustrated in the figure, the distributions induced by our methods exhibit lower peaks and heavier tails. The mean similarities of semantic entropy and semantic divergence are comparable and both are lower than those of entropy, indicating a higher degree of dispersion among generated responses. Such increased reasoning diversity encourages broader exploration of the solution space, which aligns with the observed performance gains on more challenging benchmarks.

### 5.3 Reasoning Efficiency Analysis

To investigate whether ROSE can achieve more efficient reasoning, we evaluate different values of the hyperparameter Ξ± \alpha and report the corresponding pass@8 scores and reasoning lengths. The averaged results across the four datasets are presented in Table 3 .

The results show a clear trend that increasing Ξ± \alpha reduces the reasoning length while maintaining strong pass@8 performance. In particular, moderate values of Ξ± \alpha (e.g., Ξ± = 1 \alpha=1 or Ξ± = 2 \alpha=2 ) yield the best trade-off between accuracy and efficiency, achieving higher pass@8 scores together with substantial reductions in reasoning length. Even with a relatively large value of Ξ± \alpha (i.e., Ξ± = 10 \alpha=10 ), our method still consistently outperforms GRPO variants in terms of pass@8. Overall, these results demonstrate that ROSE enables more efficient reasoning without sacrificing task performance. The evolution of response length across training steps is presented in Appendix 5.6 .

### 5.4 Ablation Study

An ablation study is conducted to analyze the contribution of each component, with the results presented in Table 4 . (1) w/o Ξ΅ \varepsilon -exploration removes the Ξ΅ \varepsilon -possibility branching mechanism (i.e., Ξ΅ \varepsilon = 0), which results in consistent performance drops across all backbone models, indicating that the model tends to fall into overly local exploration and loses exploration diversity. Additional results examining different values of Ξ΅ \varepsilon are provided in Appendix 5.7 . (2) w/ random branching randomly determines the branching positions during rollout, which leads to performance degradation, indicating that the semantic entropy metric can effectively identify uncertain points along the reasoning trajectories. (3) w/o advantage estimation removes the segment-level advantage estimation and instead directly uses the GRPO advantage formulation and loss function. This modification leads to degraded performance, which suggests that segment-level advantage estimation plays an important role in shaping learning signals during reasoning.

### 5.5 Case Study

We also conduct case studies and observe that our semantic-entropy-based approach can identify semantical uncertain positions along the reasoning paths, encouraging more diverse reasoning. Detailed examples are provided in the Appendix A .

### 5.6 Response Length Dynamics

To further investigate the effect of the hyperparameter Ξ± \alpha , we examine how the average response length per prompt during the rollout stage and the evaluation stage evolves over training steps, with the results presented in Figure 5 . We observe that as Ξ± \alpha increases, the generated response lengths in both the rollout and evaluation stages decrease substantially, and are consistently shorter than those of Dr.GRPO and GRPO. This indicates that our method can effectively regulate the generation length by adjusting Ξ± \alpha , thereby enabling more efficient reasoning.

### 5.7 Impact of Ξ΅ \varepsilon -exploration