Reinforced Efficient Reasoning via Semantically Diverse Exploration — part 3/3

Cleaned-text code points [31436, 36213). Bibliography omitted here; source.md includes it.

We investigate the impact of different values of ε \varepsilon in ε \varepsilon -exploration on model performance, with the results reported in Table 5 . When ε = 1 \varepsilon=1 , i.e., each rollout is sampled independently, our method degenerates into the Dr.GRPO algorithm. We observe that, across all backbone models, performance first improves and then degrades as ε \varepsilon increases. When ε = 0 \varepsilon=0 , exploration becomes overly local, which hinders diversity in the model’s exploration. Conversely, when ε = 1 \varepsilon=1 , the tree-structured exploration is lost, preventing effective segment-level advantage estimation.

## 6 Conclusion

In this work, we presented ROSE, a novel reinforcement learning framework designed to enhance both the reasoning accuracy and efficiency of LLMs. Specifically, to encourage more diverse reasoning exploration, our method incorporates a semantic-entropy-based branching strategy alongside an ε \varepsilon -exploration mechanism. Simultaneously, to improve efficiency, we design a length-aware segment-level advantage estimator that promotes concise reasoning paths. Extensive experiments across various mathematical benchmarks validate that ROSE significantly outperforms state-of-the-art baselines in both effectiveness and efficiency.

## Limitations

Our work mainly has two limitations. First, our experiments were conducted on models with up to 8B parameters, and we plan to investigate the scalability on larger architectures (e.g., 14B) in future work. Second, we primarily focused on mathematical reasoning tasks. In the future, we plan to extend our approach to other domains, such as code generation and question answering.

## Ethical Considerations

This work aims to enhance the reasoning capabilities of LLMs. We acknowledge that advanced reasoning abilities could potentially be misused for malicious purposes, and we advocate for the deployment of these models alongside robust safety alignment protocols to mitigate such risks. Regarding the experimental setup, all datasets utilized in this work are open-source and publicly available. We have strictly adhered to their respective licenses and ensured that our usage is consistent with their intended purposes.

## Appendix A Case Study

To more comprehensively investigate the differences between entropy-based branching and semantic-entropy–based branching, we present a case study, with the results of the two methods shown in Figure 6 and Figure 7 , respectively. Specifically, given a question, we use Llama-3.2-3B-Instruct as the backbone model to generate a complete response. We then apply two different methods to determine the branching positions based on this response and regenerate accordingly, ultimately forming a group of responses.

From Figure 6 , we observe that all responses generated by the entropy-based branching method produce incorrect answers. Notably, the shared prefix among these responses already contains an erroneous calculation (highlighted in blue). However, the branching position identified by entropy-based branching occurs after this point, which consequently propagates the incorrect reasoning into the subsequent generations.

In contrast, our proposed semantic-entropy–based branching method performs branching before the erroneous calculation (highlighted in blue), enabling more fine-grained reasoning that avoids this error. Although subsequent reasoning errors may still occur (e.g., response 2 in Figure 7 ), branching is again triggered prior to the error, ultimately yielding a correct response (e.g., response 3 in Figure 7 ). This case study demonstrates that our method can more accurately identify regions of higher uncertainty in the model’s reasoning trajectory, thereby encouraging more diverse and effective reasoning paths.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .