rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking — part 4/4

Cleaned-text code points [46839, 53567). Bibliography omitted here; source.md includes it.

Data Generation Details . As detailed in Sec. 3.4 , each round starts by self-generating step-by-step verified trajectories for 747k math word problems. The maximum tree depth d d is set to 16, with 16 MCTS rollouts conducted per problem by default. At each step, we allow to explore 8 candidate nodes, and the constant c c in Eq. 1 is set to 2 to promote greater exploration. In the bootstrap round, due to the large size of the initial policy model (236B), we used smaller parameters: 8 rollouts and 5 candidate nodes per step. To improve the accuracy of solving challenging problems in round 4, we increase the number of candidate nodes to 16 and conduct 2 MCTS tree expansions per problem using different random seeds. Detailed prompts are available in Appendix A.3 .

Training Details . In each round, we collect step-by-step verified trajectories to fine-tune the policy LLM and train the PPM. To reduce noise in synthetic math problems (e.g., incorrect ground-truth answers labeled by GPT-4), we remove synthetic problems with trajectories achieving less than 50% accuracy. Based on our extensive experiments, the policy LLM is fine-tuned from the initial base model in each round, rather than training incrementally on the model from the previous round. All policy SLMs are trained for 2 epochs with a sequence length of 4096 tokens and a batch size of 128. We use AdamW optimizer with a linear learning rate scheduler, setting the initial learning rate to 7e-6 for Qwen models, and a cosine scheduler with an initial learning rate of 5e-6 for Phi3-mini-Instruct. The PPM is trained for 1 epoch with a batch size of 512 and an initial learning rate of 7e-6.

Training the ORM and PQM . The Outcome Reward Model (ORM) and the Q-value-based Process Reward Model (PQM) share the same model architecture and training parameters with our PPM. To train the ORM, we collect trajectories from math problems containing both correct and incorrect solutions. Specifically, the two trajectories with the highest average Q-values are selected as positive examples, while the two with the lowest are chosen as negative examples. Following Qwen2.5-Math ( Yang et al., 2024 ) , we adopt the pairwise ranking loss ( Ouyang et al., 2022 ) to optimize the ORM. To train the PQM, we follow Chen et al. (2024) to use step-level Q-values as reward labels. Let 𝐱 = x ⊕ s 1 ⊕ s 2 ⊕ … ⊕ s d \mathbf{x}=x\oplus s_{1}\oplus s_{2}\oplus...\oplus s_{d} be the trajectory, with annotated Q-values 𝐐 = ( Q ⁡ ( s 1 ) , Q ⁡ ( s 1 ) , … , Q ⁡ ( s d ) ) \mathbf{Q}=(Q(s_{1}),Q(s_{1}),...,Q(s_{d})) and predicted Q-values 𝐐 ′ = ( Q ′ ​ ( s 1 ) , Q ′ ​ ( s 1 ) , … , Q ′ ​ ( s d ) ) \mathbf{Q^{\prime}}=(Q^{\prime}(s_{1}),Q^{\prime}(s_{1}),...,Q^{\prime}(s_{d})) for each step. To stabilize PQM training, we treat each trajectory as a single training sample and predict Q-values for all steps simultaneously, rather than splitting it into individual per-step samples. Specifically, to predict the Q-value Q ′ ​ ( s i ) Q^{\prime}(s_{i}) for step s i s_{i} , PQM takes the trajectory from the question up to step s i s_{i} (i.e., x ⊕ s 1 ⊕ s 2 ⊕ … ⊕ s i x\oplus s_{1}\oplus s_{2}\oplus...\oplus s_{i} ) as input and outputs a value between -1 and 1. We use a mean squared error (MSE) loss for PQM training: ℒ p ​ r ​ m ​ ( 𝐱 ) = ‖ 𝐐 − 𝐐 ′ ‖ 𝟐 \mathcal{L}_{prm}(\bf{x})=\|\bf{Q}-\bf{Q^{\prime}}\|^{2} (6)

Self-evolution Inference Costs. In the initial bootstrap round, we use DeepSeek-Coder-v2-Instruct (236B) as the policy model, using 10 nodes of 8×80GB H100 GPUs with 8 MCTS rollouts. This required approximately two weeks to finish the data generation. For rounds 2–4, using our fine-tuned 7B SLM as the policy model, data generation was performed on 15 nodes of 4×40GB A100 GPUs, with each round completed in three days. In the final round, to include more challenging problems, we increased the number of MCTS rollouts to 64, extending the data generation time to one week.

Inference Setting . In our evaluation, we run multiple MCTS to generate candidate solution trajectories. For each problem, we generate 32 candidate nodes at each step and use the PPM to score each node. Since the PPM effectively provides step-level quality evaluations, we limit MCTS to just 4 rollouts per step to update the Q-values. After completing MCTS, the trajectory with the highest PPM score is selected as the final answer. Table 9 presents the average number of tokens generated to produce a trajectory in MCTS.

Pass@N. Table 10 compares the math reasoning performance of our policy models with the instruct versions developed by the original model team. Our policy models do not consistently outperform the instruct versions. For example, on the Qwen2.5-Math-7B base model, Qwen2.5-Math-7B-Instruct achieves 4.2% higher accuracy on the MATH benchmark. However, in System 2 deep thinking paradigm, the pass@1 accuracy alone does not fully reflect the reasoning capabilities for the policy model. To provide a more comprehensive evaluation, Fig. 6 and Fig. 7 present the pass@N accuracy. In this metric, the policy model generates N N solutions under two settings: random sampling (Fig. 6 ) and PPM-augmented MCTS deep thinking (Fig. 7 ). A problem is considered solved if one solution is correct. As shown in Figure 6 , our policy model achieves comparable Pass@64 accuracy with Qwen2.5-Math-7B-Instruct on MATH and significantly outperforms it on other benchmarks. Figure 7 further compares the pass@N accuracy of our four policy models (different sizes) after MCTS deep thinking under the same PPM guidance. We can observe that after generating 64 trajectories, the pass@N accuracy of different policy models becomes comparable.

### A.2 Examples of Reasoning Trajectories

### A.3 Prompt Examples

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .