Recursive Self-Improvement via On-Policy Distillation for Reasoning — part 4/4
Cleaned-text code points [46065, 52286). Bibliography omitted here; source.md includes it.
To test whether concision can be induced directly, we add an EOS-specific auxiliary loss rather than learning from SRCL rewrites. A verified response is truncated after its final balanced boxed expression and retained as y i cut y_{i}^{\mathrm{cut}} only if the answer judge still accepts it. Each eligible response receives one EOS target: ℒ EOS ( k ) = { 1 N valid ∑ i = 1 N valid [ 1 − p θ k ( EOS ∣ x i , y i cut ) ] , N valid > 0 , 0 , N valid = 0 . \mathcal{L}_{\mathrm{EOS}}^{(k)}=\begin{cases}\dfrac{1}{N_{\mathrm{valid}}}\displaystyle\sum_{i=1}^{N_{\mathrm{valid}}}\left[1-p_{\theta_{k}}\!\left(\mathrm{EOS}\mid x_{i},y_{i}^{\mathrm{cut}}\right)\right],&N_{\mathrm{valid}}>0,\\[6.0pt] 0,&N_{\mathrm{valid}}=0.\end{cases} (11) The loss is averaged over eligible responses and set to zero when none are available. We evaluate it as an alternative to SRCL rather than as part of the main method.
#### Results.
At 8B, EOS Penalty is 2,979 tokens shorter than DCE but 4.65 points less accurate and remains below DCE+SRCL. At 4B, it reaches only 32.71% with 7,169 tokens. Direct termination pressure therefore shortens responses at a substantially larger accuracy cost than learning from concise verified rewrites.
\FloatBarrier
## Appendix I Cross-Scale Evaluation of OPSD-TTS
Table 23 reports OPSD-TTS at exact 8K and 16K output budgets. Doubling the forced budget changes Average@12 by only 0.07, 1.05, and 2.29 points at 1.7B, 4B, and 8B, respectively, and remains far below DCE. Additional generation alone therefore does not reproduce the benefit of co-evolving training.
\FloatBarrier
## Appendix J Reproducibility Details
Table 24 summarizes the data, optimization, generation, and evaluation settings used throughout the main experiments; each ablation states its deviations explicitly.
\FloatBarrier
## Appendix K Additional SRCL-Weight Sensitivity
Table 25 extends the SRCL-weight sweep to Qwen3-4B.
The sweep peaks at λ S = 35 \lambda_{S}=35 , reaching 61.88% Average@12 with 17,265 tokens. Relative to λ S = 25 \lambda_{S}=25 , it improves accuracy by 3.06 points while using 3,148 fewer tokens; larger weights then reduce accuracy, showing that the balance between guidance and concise-target learning remains important at 4B.
\FloatBarrier
## Appendix L Results on Additional Benchmarks
We additionally evaluate Qwen3-8B and Qwen3-4B under Average@12 on three benchmarks: MATH-500 ( Hendrycks et al., 2021 ) , GPQA-Diamond ( Rein et al., 2023 ) , and AMC 2023.
DCE+SRCL improves over OPSD on all three benchmarks at both scales, with the largest gains on AMC23. The improved accuracy generally accompanies longer outputs on these broader tasks, unlike the concision gains observed on the main benchmark suite.
\FloatBarrier
\FloatBarrier
## Appendix M SRCL Rewrite Filtering and Acceptance
SRCL learns only from model-generated rewrites that satisfy its concision and correctness criteria. We audit all 3,200 candidates generated during the Qwen3-8B DCE+SRCL run in table 27 . The acceptance rate remains stable between 70.63% and 73.13% across four consecutive 50-step windows, with 2,296 rewrites (71.75%) retained overall. Accepted targets are 83.74% shorter than their source rollouts, whose mean length is 1,997 tokens. Of the 904 rejected candidates, 826 fail a structural check, 76 retain explicit revision language, and two do not terminate naturally. Rejection categories record the first failed gate. A rejected rewrite contributes no SRCL loss, although its original example still receives DCE training.
#### Filtering pipeline.
A candidate is retained only if it passes all four gates below. 1. Termination and compression. The rewrite must terminate naturally, be nonempty, remain within the 12K limit, and contain fewer tokens than its source rollout.
2. Reflection-free rewriting. A case-insensitive scan rejects explicit reconsideration, correction, restart, or repeated-verification phrases, including Wait , Actually , Correction , double-check , and start over .
3. Answer validity. The rewrite must contain at least 32 tokens and a balanced boxed answer near the end. The final box must be correct, and no earlier box may contain an incorrect answer.
4. Self-containment and repetition. The rewrite may not begin as a continuation fragment, refer to omitted material, or exceed the implementation’s repeated-phrase and repeated-line thresholds.
#### Endpoint verification.
The verifier extracts the first balanced boxed answer after the final </think> delimiter, when present, and compares it with the gold answer using math_verify . If parsing fails, it falls back to case-insensitive exact matching after whitespace removal; ratio notation a : b a:b is normalized to a / b a/b . A separate structural check examines every boxed expression, requiring the final box to be correct and rejecting candidates with an earlier incorrect box.
\FloatBarrier
## Appendix N Prompt Templates
The logical prompt templates are shown below, with model-specific chat-control tokens omitted. All privileged variants use the same problem x x , verified solution g g , transition text, and on-policy response Y 0 Y_{0} ; only the message role and ordering change. SRCL uses a separate rewrite request.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .