TTPO: Test-Time Policy Optimization — part 3/3

Cleaned-text code points [31373, 41878). Bibliography omitted here; source.md includes it.
Table 7 evaluates models with thinking mode disabled to assess whether training with a thinking-mode teacher transfers reasoning capabilities to non-thinking inference. TTPO achieves dramatically larger gains than OPSD across all scales: on average, TTPO improves over the base model by +25.2 (1.7B), +30.6 (4B), and +36.4 (8B) points, while OPSD improves by only +7.1, +5.8, and +3.5 points respectively. This indicates that TTPO far more effectively absorbs the thinking teacher’s reasoning ability into the student’s non-thinking distribution. We attribute this to the asymmetric objective: the GRPO branch on negative samples directly penalizes poor reasoning patterns in the student’s own generation mode, while OPSD’s pure distillation only passively aligns the student toward the teacher without actively suppressing failure modes.

### D.3 Additional Ablations

#### RL weight λ \lambda .

As shown in Figure 8 , the raw GRPO loss is roughly an order of magnitude larger than the OPSD forward-KL loss, while the positive sample fraction increases over training as the model produces more correct answers. Without scaling, the GRPO branch would dominate gradients and destabilize training. We therefore introduce a weight λ \lambda on the GRPO loss to balance the two branches. Table 8 sweeps λ ∈ { 0.01 , 0.05 , 0.1 , 0.15 , 0.2 } \lambda\in\{0.01,0.05,0.1,0.15,0.2\} : performance peaks at λ = 0.1 \lambda{=}0.1 , which approximately equalizes the gradient magnitudes of the two branches. Both under-weighting ( λ ≤ 0.05 \lambda{\leq}0.05 , insufficient negative penalty) and over-weighting ( λ ≥ 0.15 \lambda{\geq}0.15 , excessive penalty dominating distillation) degrade results.

#### K train K_{\text{train}} positive-negative fraction.

Table 10 ablates the positive-negative composition of the K train K_{\text{train}} subset. A fixed 50/50 split outperforms both random sampling and a dynamic fraction. The dynamic strategy faces a fundamental dilemma: when the positive fraction in K K is high (i.e., pseudo-label is likely correct), proportionally reducing negative samples in K train K_{\text{train}} cancels the amplified group-relative advantages that negative samples receive — the enlarged signal is immediately diluted by fewer recipients. Conversely, if the dynamic strategy inverts the ratio (more negatives when positives dominate), it sacrifices the reliable fine-grained FKL supervision available precisely when the pseudo-label is most trustworthy, replacing it with coarser GRPO penalties. Either direction has drawbacks; a fixed 50/50 split avoids both failure modes and guarantees stable gradient contributions from both branches at every step.

#### K train K_{\text{train}} selection strategy.

Table 10 compares strategies for selecting which rollouts enter K train K_{\text{train}} . Selecting the shortest completions performs best. Since only the first 1,024 tokens participate in the gradient update, shorter trajectories ensure that these tokens constitute a larger fraction of the total reasoning chain and are more likely to contain the critical steps that determine the final answer. For longer trajectories, the first 1,024 tokens often cover only preliminary exploration, with the decisive reasoning occurring well beyond the gradient window — yielding little useful learning signal. The “Top signal” strategy selects positive samples with the highest teacher-student FKL divergence (intuitively, trajectories where the student deviates most from the teacher and thus has the most to learn) and negative samples with the highest log-probability (intuitively, confident errors that carry the strongest penalty signal). Despite this seemingly stronger per-sample signal, the strategy underperforms shortest selection — the intuition that larger divergence or higher confidence implies more useful gradients lacks theoretical grounding and appears unreliable in practice.

## Appendix E Theoretical Analysis

### E.1 Setup

For problem x x with ground-truth a ∗ a^{*} and pseudo-label a ^ \hat{a} : q t ( a ) = π θ ( ⋅ ∣ [ x ; a ] , y < t ) , p t = π θ ( ⋅ ∣ x , y < t ) , ∇ θ KL ( q t ( a ) ∥ p t ) = − ∑ v q t ( a ) ( v ) ∇ θ log p t ( v ) . \displaystyle q_{t}^{(a)}=\pi_{\theta}(\cdot\mid[x;a],y_{<t}),\quad p_{t}=\pi_{\theta}(\cdot\mid x,y_{<t}),\quad\nabla_{\theta}\mathrm{KL}(q_{t}^{(a)}\|p_{t})=-\sum_{v}q_{t}^{(a)}(v)\,\nabla_{\theta}\log p_{t}(v). (9)

### E.2 FKL Signal Analysis

#### Positive samples ( y k ∈ 𝒫 y_{k}\in\mathcal{P} , a k = a ^ a_{k}=\hat{a} ).

The trajectory agrees with teacher, so the FKL collapses to: KL ( q t ( a ^ ) ∥ p t ) | a k = a ^ = KL ( q t ∥ p t ) ⏟ thinking vs. non-thinking only . \mathrm{KL}\!\left(q_{t}^{(\hat{a})}\,\big\|\,p_{t}\right)\bigg|_{a_{k}=\hat{a}}\;=\;\underbrace{\mathrm{KL}(q_{t}\|p_{t})}_{\text{thinking vs.\ non-thinking only}}. (10) If a ^ = a ∗ \hat{a}=a^{*} this recovers standard OPSD; if a ^ ≠ a ∗ \hat{a}\neq a^{*} it reduces to answer-agnostic distillation.

#### Negative samples ( y k ∈ 𝒩 y_{k}\in\mathcal{N} , a k ≠ a ^ a_{k}\neq\hat{a} ).

The disagreement introduces a non-negative conflict term: KL ( q t ( a ^ ) ∥ p t ) | a k ≠ a ^ = KL ( q t ∥ p t ) ⏟ thinking vs. non-thinking + Δ conflict  ( t ) ⏟ ≥ 0 , \mathrm{KL}\!\left(q_{t}^{(\hat{a})}\,\big\|\,p_{t}\right)\bigg|_{a_{k}\neq\hat{a}}\;=\;\underbrace{\mathrm{KL}(q_{t}\|p_{t})}_{\text{thinking vs.\ non-thinking}}\;+\;\underbrace{\Delta_{\text{conflict}}(t)}_{\geq\,0}, (11) where Δ conflict  ( t ) \Delta_{\text{conflict}}(t) captures the teacher’s pressure to redirect reasoning from a k a_{k} toward a ^ \hat{a} . This is beneficial when a ^ = a ∗ \hat{a}=a^{*} , but harmful when a ^ ≠ a ∗ \hat{a}\neq a^{*} , particularly when a k = a ∗ a_{k}=a^{*} , as the gradient actively suppresses correct reasoning.

#### Method comparison.

By Eq. 10 , FKL on 𝒫 \mathcal{P} is at worst benign and at best recovers standard OPSD; GRPO on 𝒩 \mathcal{N} is label-agnostic. U-OPSD applies FKL to 𝒩 \mathcal{N} instead, where Δ conflict \Delta_{\text{conflict}} (Eq. 11 ) misdirects correct trajectories whenever a ^ ≠ a ∗ \hat{a}\neq a^{*} , and requires reliable pseudo-labels since low consensus makes Δ conflict \Delta_{\text{conflict}} harmful while unanimous consensus yields waste of rollouts ( | 𝒩 | = 0 |\mathcal{N}|=0 ).

### E.3 Majority-Vote vs. Ground-Truth Routing

With binary reward r k = 𝟏 [ a k ∈ majority ] r_{k}=\mathbf{1}[a_{k}\in\text{majority}] , r ¯ = | 𝒫 | / K \bar{r}=|\mathcal{P}|/K . The GRPO advantage for k ∈ 𝒩 k\in\mathcal{N} : A k = − | 𝒫 | / K 1 − | 𝒫 | / K , k ∈ 𝒩 . A_{k}=-\sqrt{\frac{|\mathcal{P}|/K}{1-|\mathcal{P}|/K}},\quad k\in\mathcal{N}. (12)

#### GT routing

( 𝒫 GT = { k : a k = a ∗ } \mathcal{P}_{\text{GT}}=\{k:a_{k}=a^{*}\} ). On hard problems with small | 𝒫 GT | |\mathcal{P}_{\text{GT}}| , both branches vanish: | 𝒫 GT | K ≈ 0 \displaystyle\frac{|\mathcal{P}_{\text{GT}}|}{K}\approx 0\quad and ∑ k ∈ 𝒫 GT ∇ θ KL ( q t ∥ p t ) ⏟ FKL ≈ 𝟎 , \displaystyle\text{and}\quad\underbrace{\sum_{k\in\mathcal{P}_{\text{GT}}}\nabla_{\theta}\mathrm{KL}(q_{t}\|p_{t})}_{\text{FKL}}\approx\mathbf{0}, (13) | A k | = | 𝒫 GT | / K 1 − | 𝒫 GT | / K ≈ 0 \displaystyle|A_{k}|=\sqrt{\frac{|\mathcal{P}_{\text{GT}}|/K}{1-|\mathcal{P}_{\text{GT}}|/K}}\approx 0\quad and A k  ∇ θ  log ⁡ π θ  ( y k ∣ x ) ⏟ GRPO ≈ 𝟎 . \displaystyle\text{and}\quad\underbrace{A_{k}\,\nabla_{\theta}\log\pi_{\theta}(y_{k}\mid x)}_{\text{GRPO}}\approx\mathbf{0}. (14)

#### Vote routing

( 𝒫 vote = { k : a k = a ^ } \mathcal{P}_{\text{vote}}=\{k:a_{k}=\hat{a}\} ). The model always forms a non-empty consensus | 𝒫 vote | > 0 |\mathcal{P}_{\text{vote}}|>0 , so both branches remain active: | 𝒫 vote | K > 0 \displaystyle\frac{|\mathcal{P}_{\text{vote}}|}{K}>0\quad and ∑ k ∈ 𝒫 vote ∇ θ KL ( q t ∥ p t ) ⏟ FKL ≠ 𝟎 , \displaystyle\text{and}\quad\underbrace{\sum_{k\in\mathcal{P}_{\text{vote}}}\nabla_{\theta}\mathrm{KL}(q_{t}\|p_{t})}_{\text{FKL}}\neq\mathbf{0}, (15) A k = − | 𝒫 vote | / K 1 − | 𝒫 vote | / K < 0 \displaystyle A_{k}=-\sqrt{\frac{|\mathcal{P}_{\text{vote}}|/K}{1-|\mathcal{P}_{\text{vote}}|/K}}<0\quad and A k  ∇ θ  log ⁡ π θ  ( y k ∣ x ) ⏟ GRPO ≠ 𝟎 . \displaystyle\text{and}\quad\underbrace{A_{k}\,\nabla_{\theta}\log\pi_{\theta}(y_{k}\mid x)}_{\text{GRPO}}\neq\mathbf{0}. (16) As the model improves, a ^ \hat{a} converges toward a ∗ a^{*} (Figure 6 ).

## Appendix F Limitations and Future Work

#### Dependence on majority-vote quality.

TTPO relies on majority voting to generate pseudo-labels and classify positive/negative samples. When the sample budget K K is very small or the problem is so difficult that no rollout produces a correct answer, the voting signal degrades and both branches receive noisy supervision. Adaptive strategies that adjust the positive-negative ratio or fall back to pure RL under low-consensus conditions could mitigate this.

#### Domain scope.

Our experiments are restricted to mathematical reasoning with verifiable final answers. Extending TTPO to domains where correctness is harder to extract automatically — such as code generation (requiring execution-based verification) or open-ended reasoning (requiring learned reward models) — remains unexplored.

#### Dynamic training curriculum.

TTPO applies a fixed asymmetric objective throughout training. As the model improves and pseudo-label accuracy rises, the optimal balance between distillation and RL may shift. A curriculum that dynamically adjusts the RL weight or positive-negative fraction to training dynamics could further improve efficiency.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .