Reinforcement Learning via Self-Distillation — part 6/6
Cleaned-text code points [78711, 83670). Bibliography omitted here; source.md includes it.
### D.2 Learning with rich environment feedback
#### D.2.1 Additional Results
Figure 15 shows the average accuracy of SDPO and GRPO stratified by question difficulty. LCB differentiates between easy, medium, and hard questions. As displayed, SDPO significantly improves over GRPO in solving medium and hard questions, highlighting the importance of rich feedback for challenging tasks. Note that this categorization of questions is different from the one in Section 5 .
In Figure 16 , we compare different train batch sizes and number of rollouts for training GRPO and SDPO on LCBv6.
Complementing the results shown in Figure 8 , we show additional results using Qwen2.5-Instruct ( Qwen et al., 2024 ) in Figure 17 .
#### D.2.2 Training Stability
Figure 18 shows diverse metrics logged during training, including the loss, entropy, average gradient norm, and average response length.
.
#### D.2.3 Baselines
Table 9 compares the performance on LCBv6 of various baselines, including two variants of GRPO, GSPO, and CISPO to SDPO.
### D.3 Test-time self-distillation
Complementing the results shown in Section 5 , we show the discovery@ k k curves for all hard question in Figure 20 , and report the mean number of generations until the first discovery in Table 10 . Further, Table 11 shows the per-question accuracy of the self-teacher at the initial training step of SDPO. In Figure 19 , we ablate the choice of batch size for SDPO and the in-context reprompting strategy for multi-turn sampling.
In the selection of hard questions, we have discarded one malformed question (Q9) where the coding environment did not correctly validate the solution due to rounding inaccuracies, which led to failures even with correct logic.
## Appendix E Experiment Details
### E.1 Technical setup
All experiments were conducted on a single node equipped with four NVIDIA GH200 GPUs, for a total of 378GB VRAM. Our environment is built on top of the NVIDIA PyTorch container nvcr.io/nvidia/pytorch:25.02-py3 , with CUDA 12.8 and PyTorch v2.7.0.
Our implementation is based on the verl library ( Sheng et al., 2025 ) . We use PyTorch Fully Sharded Data Parallel (FSDP2) for distributed training. For rollout generation, we employ vLLM ( Kwon et al., 2023 ) , which enables efficient batched inference on the multi-GPU node.
### E.2 Hyperparameters
We summarize hyperparameters used for SDPO in Table 12 and those used for GRPO in Table 13 .
#### E.2.1 Details on Hyperparameter Selection ( Section 3 )
For GRPO in the experiments in Section 3 , we perform a grid search over learning rates { 10 − 5 , 10 − 6 } \{10^{-5},10^{-6}\} and minibatch sizes { 8 , 32 } \{8,32\} . For on-policy GRPO, we search over the same learning rates while fixing the minibatch size to 32. For SDPO, we grid-search over KL variants (forward KL, Jensen–Shannon), learning rates { 10 − 5 , 10 − 6 } \{10^{-5},10^{-6}\} , and minibatch sizes { 8 , 32 } \{8,32\} . For each method (GRPO, on-policy GRPO, and SDPO), we select a single hyperparameter configuration that achieves the highest validation accuracy within the first 5 hours of training, evaluated across all datasets and models used in Section 3 . We further report results obtained by selecting the optimal hyperparameter configuration separately for each model and dataset in Table 3 .
### E.3 User Templates
For multiple-choice questions and tool use, the model must be prompted in a task-specific manner. We therefore provide the prompt templates used for these settings below.
## Appendix F Qualitative Examples
### F.1 Visualization of Advantages
Figure 21 compares the advantages of SDPO and GRPO in a representative example.
### F.2 Examples
Below, we show an example from training SDPO on LCBv6 using Qwen3-8B.
⬇
### F.3 Environment Feedback
We show three examples of feedback in our coding environment, inspired by LeetCode.
### F.4 Illustrative Example
Figure 22 shows an illustrative example of the dense credit assignment in SDPO.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .