TTPO: Test-Time Policy Optimization — part 2/3

Cleaned-text code points [15993, 31373). Bibliography omitted here; source.md includes it.

## 4 Experiments

### 4.1 Experimental Setup

#### Implementation.

We evaluate on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B ( Yang et al., 2025 ) , all fine-tuned with LoRA ( r = 64 r{=}64 , α = 128 \alpha{=}128 ) on all linear layers. We consider two settings: (1) OpenThoughts setting , where models are trained on labeled data but TTPO does not use the labels — they serve only for comparison with label-dependent baselines; and (2) TTT setting , where models are trained directly on the test set without any annotations. The shared training configuration follows OPSD, and full training configurations are provided in Appendix A .

#### Baselines.

We compare against: (1) OPSD ( Zhao et al., 2026 ) , on-policy self-distillation with ground-truth labels; (2) GRPO ( Shao et al., 2024 ) , RL with ground-truth rewards; (3) TTRL ( Zuo et al., 2026 ) , label-free RL via majority-vote rewards; and (4) OPSD-TTT , self-distillation using the model’s temperature-0 output under thinking mode as privileged information.

#### Evaluation.

We evaluate on five competition-level math benchmarks: AIME 2025, AIME 2026, HMMT 2025, HMMT 2026, and BRUMO 2025 ( Dekoninck et al., 2026 ) . To ensure train-inference consistency, evaluation is performed with thinking mode enabled (non-thinking evaluation in Appendix D.2 ). All results are reported as Avg@12 with temperature 1.0. For OPSD and TTPO, we train for 100 steps and report the peak performance across checkpoints saved every 25 steps. For GRPO and TTRL, we train for 500 steps and report the peak across all checkpoints.

#### TTPO-specific hyperparameters.

We sample K = 64 K{=}64 trajectories per problem to ensure reliable majority voting on hard problems with low pass rates, with a maximum generation length of 16,000 tokens to avoid truncation that prevents answer extraction. From the K K rollouts, K train = 8 K_{\text{train}}{=}8 are selected (50% positive, 50% negative) for the gradient update, and the RL weight λ = 0.1 \lambda{=}0.1 balances gradient magnitudes between the two branches (ablated in Appendix D.3 ).

### 4.2 Main Results

#### Labeled training data.

Table 1 compares methods trained on OpenThoughts, where OPSD and GRPO use ground-truth labels while TTPO relies solely on majority-vote pseudo-labels. TTPO exceeds the label-dependent OPSD across all three model scales (40.1 vs. 39.7 on 1.7B, 58.6 vs. 58.4 on 4B, 62.6 vs. 61.7 on 8B in average), despite without ground-truth supervision. This demonstrates that majority-vote pseudo-labels, when combined with our asymmetric objective, can substitute for ground-truth annotations without sacrificing performance. The improvements are consistent across scales, and notably, TTPO on Qwen3-4B (58.6 avg) already matches the Qwen3-8B base model (58.6 avg), suggesting that our training recipe effectively amplifies a smaller model’s reasoning capacity to the level of a 2 × 2\times larger untrained model.

#### Label-free test-time training.

Table 2 evaluates the purely label-free TTT setting where models train directly on the test problems. TTPO consistently and substantially outperforms both TTRL and OPSD-TTT across all model scales. On Qwen3-1.7B, TTPO achieves 45.2 average — +3.3 over OPSD-TTT and +5.4 over TTRL, representing a 7.2-point absolute gain over the base model. The gap over TTRL demonstrates the value of dense distributional guidance: while TTRL provides only binary reward signals, TTPO additionally leverages the answer-conditioned teacher to transfer token-level knowledge on correct trajectories. The gap over OPSD-TTT — which uses deterministic (greedy decoding with thinking mode enabled) answers as privileged information rather than majority-vote pseudo-labels — shows that even with a reasonable self-distillation baseline, our asymmetric design extracts substantially more signal by additionally exploiting negative samples through selective RL penalties. Cross-scale comparison further highlights the efficiency: TTPO on Qwen3-4B (61.1 avg) already surpasses Qwen3-8B base (60.7 avg), demonstrating that label-free test-time training with TTPO can close the gap between model sizes.

### 4.3 Ablation Studies

#### Token-level selection.

Table 3 isolates the contribution of each token-level selection mechanism. Both components improve over uniform updates, but their effects are complementary and target different failure modes: removing positive-sample weighting (w/o pos. weight) uniformly distills all tokens including low-value (low-entropy, low-divergence) positions where the student has already converged, diluting the gradient signal from genuinely informative tokens; removing negative-sample masking (w/o neg. mask) penalizes all tokens indiscriminately — not only causing collateral damage to locally correct reasoning steps that cannot be offset without positive-advantage updates, but also allowing anomalous (low-probability, low-entropy) tokens to dominate gradient updates, injecting substantial noise into optimization. The full method benefits from both — focusing distillation where it matters and penalizing only where errors originate.

#### Update strategy.

Figure 3 compares update strategy combinations. The full TTPO (pos=FKL, neg=GRPO, 48.9) substantially outperforms all alternatives. FKL is well-suited to positive samples because their answers match the pseudo-label by definition: even when the label is wrong, the teacher is conditioned on the same answer the student produced, reducing to thinking-to-non-thinking distillation (45.7) that safely transfers careful reasoning. Hence positive-only FKL (46.7) outperforms all-FKL (46.3) and negative-only FKL (43.9), which forces the teacher to steer unmatched answers and injects corrupted signals. GRPO is better suited to negative samples: although its credit assignment is coarser, on hard TTT data most negatives are correctly identified (answer ≠ \neq pseudo-label ∧ \land ≠ \neq ground truth), so a label-free penalty is strictly safer than corrupted distillation. GRPO on positives (37.2) lacks this robustness—it directly reinforces trajectories, and wrong pseudo-labels reverse the update with no mitigation. The reversed assignment (pos=GRPO, neg=FKL, 37.2) therefore performs worst, combining brittle reinforcement on positives with corrupted distillation on negatives.

### 4.4 Analysis

#### Privileged information.

Table 4 ablates privileged information under different teacher modes. With a thinking-mode teacher, the teacher–student distributional gap is already large, so a short answer suffices as a lightweight hint that steers the teacher without crowding out its reasoning. Even a wrong pseudo-label remains consistent with the student’s answer in positive samples, and the update degenerates into thinking-to-non-thinking distillation—still a well-posed and beneficial signal (46.5 vs. 45.8). A full trajectory, by contrast, dominates the context and reduces the teacher to completing a given prefix rather than reasoning independently, degrading performance (41.1). With a non-thinking teacher, the gap is inherently small: a short answer barely shifts the distribution (33.6), while a full trajectory supplies needed context (40.8) but leaves both sides under weak reasoning, making training highly sensitive to pseudo-label noise. The thinking-teacher + answer setting thus offers the best trade-off between guidance and robustness.

#### Generalization beyond the target task.

To verify that TTPO acquires generalizable reasoning improvements rather than overfitting to specific problems, we train on each benchmark separately and evaluate on all three (Figure 4 ). Models trained on any single benchmark consistently improve on the other two as well. This cross-benchmark transfer confirms that TTPO strengthens underlying reasoning capabilities rather than memorizing problem-specific patterns.

#### Upper bound with labeled supervision.

We replace majority-vote pseudo-labels with ground-truth answers to probe the performance ceiling (Figure 5 ). Surprisingly, TTPO with pseudo-labels outperforms both TTPO w/ GT and OPSD (Leakage). First, perfectly correct labels are hard to match on difficult problems, yielding few or zero positives per instance; this starves the FKL branch and leaves GRPO with near-zero advantages that barely penalize negatives (Figure 8 ). Majority-vote labels, being easier to match, keep a healthy positive–negative split and both branches active. Second, AIME26 ground-truth answers are short numbers that barely shift the thinking teacher, unlike the richer OpenThoughts trajectories—reliable, yet too brief to guide strongly. The entropy plot (right) corroborates this: TTPO with pseudo-labels sustains higher entropy, as majority voting and distillation jointly promote exploration that compensates for—and ultimately surpasses—the theoretical benefit of perfect labels.

#### Sustainable self-evolution.

Since majority voting generates the training signal, the base model’s Maj@12 sets the initial ceiling on pseudo-label quality. We track Avg@12 and Maj@12 throughout training to examine whether TTPO can break this ceiling (Figure 6 ). As training progresses, Avg@12 rises steadily to the base Maj@12, confirming that the collective knowledge in majority voting is distilled into single-sample performance. More importantly, Maj@12 does not stagnate but rises in tandem: as the model improves, higher-quality rollouts yield more accurate pseudo-labels, which in turn raise the training ceiling for later steps. This self-evolving cycle enables TTPO to improve beyond its initial supervision and ultimately outperform training with ground-truth that exceed the model’s current capacity (Figure 5 , left).

## 5 Conclusion

We introduced TTPO, which brings OPSD into label-free test-time training by combining it with RL under an asymmetric objective. Distillation provides dense token-level guidance on positives, while RL supplies a robust, label-free penalty on negatives. With token-level selection on both branches, TTPO matches ground-truth-supervised methods, substantially outperforms existing label-free approaches, and exhibits self-evolution with strong cross-task generalization.

## Appendix A Implementation Details

We provide complete training and evaluation configurations in Tables 5 and 6 . All experiments use the AdamW ( Loshchilov & Hutter, 2017 ) optimizer with bfloat16 precision and Flash Attention 2. TTPO experiments use 4 × \times H20 GPUs; all other methods use 8 × \times H20 GPUs. We adopt full-vocabulary logit distillation for all distillation-based methods (OPSD, OPSD-TTT, and TTPO). Following Zhao et al. (2026) , we use a thinking-mode-off student / thinking-mode-on teacher configuration, and the teacher is fixed to base model weights (LoRA adapters disabled) throughout training. For TTPO and OPSD-TTT, we set the maximum sampling length to 16,000 tokens to reduce answer extraction failures caused by truncation during majority voting, while the gradient update still only applies to the first 1,024 completion tokens.

## Appendix B Prompt Templates

We list the prompt templates used for both the student and teacher. All prompts are wrapped with the model’s chat template via apply_chat_template . When privilege_info = none (the no-privilege ablation in Table 4 ), the teacher prompt is identical to the student prompt but with thinking mode enabled.

## Appendix C Case Study: Token-Level Selection Visualization

We visualize how our token weighting (§ 3.3 ) and token masking (§ 3.4 ) operate on a concrete example. Excerpts are drawn from a positive sample and a negative sample generated for the same geometry problem. 1 1 1 Problem: Given square A ​ B ​ C ​ D ABCD with L L on C ​ D CD , K K on extension of D ​ A DA beyond A A , ∠ ​ K ​ B ​ L = 90 ∘ \angle KBL=90^{\circ} , K ​ D = 19 KD=19 , C ​ L = 6 CL=6 . Find L ​ D LD . Ground truth: 7 7 . The positive sample reaches the correct answer 7 7 ; the negative arrives at 397 \sqrt{397} . Recall that token weighting down-weights low-entropy, low-divergence positions where the student has converged, while token masking suppresses low-probability, low-entropy positions representing anomalous model outputs.

### C.1 Token Weighting

The contrast is stark: coordinate values (“ ( 0 , a ) (0,a) ”, “ ( 0 , 0 ) (0,0) ”, “ x = 0 x=0 ”) receive near-zero weight—these are deterministic once the setup is chosen, and both student and teacher assign near-unit probability to each digit. High weight concentrates on geometric insights that determine the solution strategy: “the extension beyond A A is the line… going downwards” (identifying the geometric locus) and “So K K is at ( 0 , y ) (0,y) , for some y < 0 y<0 ” (drawing the conclusion). These are positions where the student is uncertain which geometric fact to invoke or the teacher favors a different continuation, making them the sole source of learning signal. Token weighting thus focuses distillation on where to reason and what to conclude , not on reproducing mechanical substitutions the model already handles reliably.

### C.2 Token Masking

The kept tokens are precisely the positions worth penalizing: local errors (“which is vertical”— C ​ D CD is actually horizontal under the model’s own coordinates), context-inconsistent expressions (“we move… units from C C along C ​ D CD ”—applying a vertical displacement to a horizontal segment), and the resulting anomalous conclusion (“So, L = L= ”—committing to coordinates that place L L off the intended side). Masked-out tokens, by contrast, are locally correct arithmetic (“ ( s , s ) (s,s) ”, “ = 6 =6 ”) and formatting that would appear identically in a correct solution; penalizing them would damage valid computation skills without addressing the actual reasoning flaw. Token masking thus restricts the penalty gradient to the confident errors that cause failure —wrong geometric claims and their immediate consequences—while leaving shared, reusable sub-skills intact.

### C.3 Complementarity

The two mechanisms implement a dual philosophy: token weighting asks “ where does the model still need to learn? ” and suppresses already-converged positions in positive samples; token masking asks “ where is the model confidently wrong? ” and suppresses locally-correct or uncertain positions in negative samples. Both avoid wasting gradient on low-signal tokens—from opposite directions—yielding more efficient and stable training.

## Appendix D Additional Experiments

### D.1 Training Dynamics

Figure 7 compares training dynamics of Qwen3-1.7B TTT on AIME 2026 across methods. The most striking observation is that TTPO w/ GT exhibits dramatically weaker training signal than all other methods: its loss barely decreases and frequently stagnates near zero. This directly validates our theoretical analysis (Eq. 13 – 14 ) — on hard AIME problems where | 𝒫 GT | ≈ 0 |\mathcal{P}_{\text{GT}}|\approx 0 , GT routing starves both branches simultaneously. In contrast, TTPO with majority-vote pseudo-labels maintains a steady and substantial loss decrease throughout training, confirming that vote-based routing keeps both branches active. OPSD (Leakage) and OPSD-TTT both show consistent loss reduction; notably, TTPO achieves comparable or stronger training dynamics despite operating in a fully label-free setting.

### D.2 Non-Thinking Evaluation