# TTPO: Test-Time Policy Optimization

###### Abstract

Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO) , an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization. Our code is available at https://github.com/ZJU-REAL/TTPO .

## 1 Introduction

Large language models (LLMs) have achieved remarkable mathematical reasoning through extended chain-of-thought generation ( Team et al., 2026 ; Xu et al., 2026a ; Zeng et al., 2026 ; Singh et al., 2025 ; Team, 2026 ) , largely powered by post-training with reinforcement learning from verifiable rewards (RLVR) ( Shao et al., 2024 ; Yu et al., 2026 ; Guo et al., 2025 ) . Yet such outcome rewards are inherently coarse, broadcasting a single sequence-level scalar uniformly across all tokens and leaving the reasoning at each step unsupervised.

A complementary line supplies dense, token-level supervision: on-policy self-distillation (OPSD) ( Zhao et al., 2026 ) conditions the same policy on the ground-truth answer to form a teacher that re-scores the student’s own rollouts token by token ( Ye et al., 2026 ; Yang et al., 2026b ) . Recent work increasingly combines the two signals, through auxiliary distillation losses, credit redistribution, or routing between them ( Yang et al., 2026a ; Liu et al., 2026b ; Lu et al., 2026a ; Han et al., 2026 ; Li et al., 2026a ) . All these methods, however, assume ground-truth answers: the reward needs them for verification, and the teacher needs them as privileged context. In test-time training (TTT) ( Sun et al., 2020 ) , where a model improves on the very problems it must solve and labels never arrive, none of these methods applies.

Without labels, supervision must come from the model itself: sample a group of rollouts for each problem, and take the majority answer as a pseudo-label. TTRL ( Zuo et al., 2026 ) uses this pseudo-label as a reward and improves reasoning without any labels, but the reward is still one scalar per trajectory, and when the majority is wrong, training reinforces the error ( Lin et al., 2026 ) . The natural next step is to let the pseudo-label replace the ground-truth answer in OPSD, which has been tried in two forms: distilling all rollouts toward the pseudo-label-conditioned teacher ( Gkountouras et al., 2026 ) , or distilling only the rollouts that disagree with the pseudo-label ( Li et al., 2026b ) . But dense supervision magnifies label errors: a corrupted reward misleads once per trajectory; a corrupted teacher misleads at every token .

These errors are the common case: on competition-level problems, the pseudo-label is wrong for ∼ {\sim} 85% of prompts (Figure 1 , a). Learning from such a label seems infeasible. However, even when the pseudo-label is wrong, ∼ {\sim} 79% of the rollouts that disagree with it are wrong too. A penalty on a disagreeing rollout is therefore usually correct whether or not the pseudo-label is, because it uses the disagreement alone, never the pseudo-label’s answer. Distillation toward the pseudo-label has no such tolerance: a wrong answer enters the teacher and misleads every token. The same asymmetry underlies negative learning from noisy labels, where stating what a sample is not remains reliable even when the label is wrong ( Kim et al., 2019 ) .

We propose Test-Time Policy Optimization (TTPO), which applies each signal where it is reliable: GRPO penalties on the rollouts that disagree with the pseudo-label, and OPSD distillation on the rollouts that agree with it. The distillation branch tolerates wrong pseudo-labels for a different reason: the teacher is conditioned on the answer that the agreeing rollouts themselves produced, so even when that answer is wrong, the update distills the model’s thinking mode into its non-thinking mode rather than toward an arbitrary error. Finally, token-level selection sharpens both branches, weighting distillation toward positions the student has not yet mastered and masking penalties to the confident errors that caused the failure. By combining both signals, our asymmetric design not only makes more effective use of both positive and negative rollouts while remaining robust to pseudo-label noise, but is also naturally calibrated to the model’s current capability, enabling a virtuous cycle of self-evolution that ground-truth routing cannot sustain (Figure 5 , 6 ).

Trained without any labels, TTPO matches or exceeds label-supervised OPSD across Qwen3-1.7B/4B/8B on five competition-level benchmarks, and in the pure TTT setting raises the 1.7B base model from 38.0% to 45.2% average accuracy, ahead of both TTRL and self-distillation baselines. With thinking mode disabled, the gains reach +25.2% to +36.4% across scales, several times the gain of label-supervised OPSD. We further validate that training on any one benchmark improves the other two, indicating generalizable reasoning rather than problem-specific overfitting (Figure 4 ). Our contributions are: 1. We show that majority-vote pseudo-labels remain useful despite frequent errors: though wrong on ∼ {\sim} 85% of competition-level prompts, ∼ {\sim} 79% of disagreeing rollouts are wrong too, so penalizing disagreement stays correct while distillation does not.

2. We propose TTPO, which applies each signal where it stays correct: agreeing rollouts are distilled toward an answer-conditioned teacher, disagreeing rollouts receive GRPO penalties, with token-level selection in both branches.

3. Trained without any labels, TTPO matches or exceeds label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, and further demonstrates strong cross-task generalization.

## 2 Related Work

### 2.1 Test-Time Training for Reasoning

Test-time training (TTT) adapts models to unlabeled test data at inference time ( Sun et al., 2020 ; Li et al., 2026b ; Du et al., 2025 ) . TTRL ( Zuo et al., 2026 ) extends TTT to LLM reasoning by sampling multiple trajectories per problem, deriving pseudo-rewards via majority voting, and training with GRPO ( Shao et al., 2024 ) . Follow-up work addresses TTRL’s sensitivity to consensus quality: Hi-TTRL ( Xu et al., 2026b ) introduces hierarchical reward shaping with hints, while SCRL ( Yan et al., 2026 ) applies selective pseudo-labeling to filter unreliable majorities. However, these methods remain purely RL-based, propagating a single sequence-level reward uniformly across all tokens.

### 2.2 On-Policy Self-Distillation

On-policy distillation trains a policy on its own rollouts under a teacher ( Agarwal et al., 2024 ; Gu et al., 2026 ; Wen et al., 2023 ) . Recent self-distillation variants remove the need for a separate teacher by conditioning the same model on privileged information available only during training ( Zhao et al., 2026 ; He et al., 2026 ; Lu et al., 2026b ) . Several studies further incorporate the resulting teacher–student log-probability gap into RLVR, either as advantage scaling ( Yang et al., 2026a ) , a detached auxiliary objective ( Lu et al., 2026a ) , a routing mechanism ( Han et al., 2026 ; Li et al., 2026a ) , or reward-densifying local supervision ( Xu et al., 2026c ; Ye et al., 2026 ; He et al., 2026 ) . We follow this idea and introduce an asymmetric objective that decouples the treatment of positive and negative samples to tolerate pseudo-label errors.

### 2.3 Token-Level Weighting and Masking

Recent work recognizes that not all tokens merit equal gradients during training ( Xiao et al., 2026 ) . In distillation, TIP ( Xu et al., 2026c ) shows that training on fewer than 10% of tokens, selected by student entropy and teacher–student divergence, nearly matches full-token performance. In RL, STAPO ( Liu et al., 2026a ) masks spurious low-probability, low-entropy tokens in positive samples that receive disproportionate reward gradients, while Wu et al. (2026) address the erroneous penalization of locally correct tokens within failed trajectories through reward recalibration. TTPO applies token-level selection to both branches of its objective: down-weighting converged positions in the distillation branch and selectively penalizing only confident errors in the RL branch.

## 3 Method

### 3.1 Preliminaries and Problem Setup

Let π θ \pi_{\theta} denote the language model and { x i } i = 1 N \{x_{i}\}_{i=1}^{N} be a set of test-time problems without ground-truth labels. For each problem x x , we sample K K trajectories { y 1 , … , y K } ∼ π θ ( ⋅ ∣ x ) \{y_{1},\ldots,y_{K}\}\sim\pi_{\theta}(\cdot\mid x) and extract final answers a k = Extract ⁡ ( y k ) a_{k}=\operatorname{Extract}(y_{k}) .

#### Majority-vote pseudo-labeling.

We cluster answers by mathematical equivalence, select the largest cluster as the pseudo-label a ^ \hat{a} with consensus count c = | { k : a k ≡ a ^ } | c=|\{k:a_{k}\equiv\hat{a}\}| , and partition trajectories into positive samples 𝒫 = { k : a k ≡ a ^ } \mathcal{P}=\{k:a_{k}\equiv\hat{a}\} that agree with pseudo-label and negative samples 𝒩 = { k : a k ≢ a ^ } \mathcal{N}=\{k:a_{k}\not\equiv\hat{a}\} that disagree.

#### Answer-conditioned teacher.

Following OPSD ( Zhao et al., 2026 ) , we construct a teacher by conditioning the same model on the pseudo-label a ^ \hat{a} as privileged information. Given shared completion tokens y k y_{k} , the teacher and student differ only in their prompt prefixes: q t ( a ^ ) \displaystyle q_{t}^{(\hat{a})} = π θ ( ⋅ ∣ [ x ; a ^ ] teacher , y < t ) , (no grad) \displaystyle=\pi_{\theta}(\cdot\mid[x;\hat{a}]_{\text{teacher}},y_{<t}),\quad\text{(no grad)} (1) p t \displaystyle p_{t} = π θ ( ⋅ ∣ x student , y < t ) . (with grad) \displaystyle=\pi_{\theta}(\cdot\mid x_{\text{student}},y_{<t}).\quad\text{(with grad)} (2)

### 3.2 Motivation: Why Asymmetric?

In the TTT setting, the training set is the test set — consisting of competition-level problems that are inherently difficult for the model. As a result, majority-vote pseudo-labels are frequently wrong: on AIME 2026 with Qwen3-1.7B, we observe that pseudo-labels are wrong for ∼ {\sim} 85% of prompts on average (Figure 1 , a). Naively applying self-distillation to all trajectories using the corrupted teacher would propagate errors to every sample.

However, we observe a key structural property: even when a ^ ≠ a ∗ \hat{a}\neq a^{*} , ∼ {\sim} 79% of negative samples produce answers that are neither a ^ \hat{a} nor a ∗ a^{*} — penalizing them is correct regardless of pseudo-label quality. This motivates an asymmetric design that minimizes the blast radius of pseudo-label errors: • Pure FKL on all samples : error propagates to all K K trajectories (the teacher distribution is entirely conditioned on wrong a ^ \hat{a} ).

• Asymmetric (FKL for 𝒫 \mathcal{P} , GRPO for 𝒩 \mathcal{N} ) : only the small | 𝒫 | |\mathcal{P}| set is corrupted. GRPO on 𝒩 \mathcal{N} requires only “not in the majority cluster” — independent of a ^ \hat{a} ’s content — and is correct for the vast majority of negative samples.

### 3.3 Positive Samples: OPSD Branch

For each positive sample y k ∈ 𝒫 y_{k}\in\mathcal{P} , we apply OPSD’s forward KL with per-token weighting: ℒ OPSD ( k ) = 1 T k ∑ t = 1 T k w ( t ) ⋅ KL ( q t ( a ^ ) ∥ p t ) , \mathcal{L}_{\text{OPSD}}(k)=\frac{1}{T_{k}}\sum_{t=1}^{T_{k}}w(t)\cdot\text{KL}\left(q_{t}^{(\hat{a})}\,\|\,p_{t}\right), (3) where T k T_{k} is the number of valid response tokens and w ⁡ ( t ) w(t) is the token weight described below.

#### Token weighting.

Not all tokens offer equal learning value. Inspired by the token importance analysis of Xu et al. (2026c) , we design a weighting scheme that down-weights positions where the student has already converged. We measure two complementary signals — student entropy H ⁡ ( t ) H(t) and teacher-student divergence Δ ( t ) = KL ( q t ∥ p t ) \Delta(t)=\text{KL}(q_{t}\|p_{t}) — normalize each to [ 0 , 1 ] [0,1] via per-sample min-max normalization, and combine them with a Soft-OR: w ⁡ ( t ) = H ^ ​ ( t ) + Δ ^ ​ ( t ) − H ^ ​ ( t ) ⋅ Δ ^ ​ ( t ) . w(t)=\hat{H}(t)+\hat{\Delta}(t)-\hat{H}(t)\cdot\hat{\Delta}(t). (4) This assigns high weight when at least one signal indicates learning value (the student is uncertain, or confidently wrong), and approaches zero only when both signals are low — the student is already confident and aligned with the teacher.

### 3.4 Negative Samples: GRPO Branch

For negative samples y k ∈ 𝒩 y_{k}\in\mathcal{N} , we apply GRPO ( Shao et al., 2024 ) . Each trajectory receives a binary reward based on majority-vote classification: r k = 1 r_{k}=1 if its answer matches the pseudo-label, r k = 0 r_{k}=0 otherwise. Advantages are computed group-relatively over all K K rollouts per problem, so that A k < 0 A_{k}<0 for negative samples, yielding a penalty: ℒ GRPO ( k ) = − A k T k ∑ t = 1 T k m ( t ) ⋅ log π θ ( y k ( t ) ∣ x , y k ( < t ) ) . \mathcal{L}_{\text{GRPO}}(k)=-\frac{A_{k}}{T_{k}}\sum_{t=1}^{T_{k}}m(t)\cdot\log\pi_{\theta}(y_{k}^{(t)}\mid x,y_{k}^{(<t)}). (5)

In standard GRPO, positive-advantage gradients counterbalance erroneous penalties on locally correct tokens within failed trajectories. In our asymmetric framework, GRPO acts exclusively on negative samples, removing this counterbalance. This False Penalties on Negative Samples problem ( Wu et al., 2026 ) necessitates active reduction of collateral damage.

#### Token masking.

Drawing on the insight from Liu et al. (2026a) that low-probability, low-entropy tokens represent anomalous model behavior, we design a masking scheme to identify tokens most responsible for errors. We score each token by: s ( t ) = − log π θ ( y k ( t ) ∣ x , y k ( < t ) ) ⋅ ( 1 − H ^ ( t ) ) , s(t)=-\log\pi_{\theta}(y_{k}^{(t)}\mid x,y_{k}^{(<t)})\cdot(1-\hat{H}(t)), (6) where the raw negative log-probability serves as the dominant ranking factor and ( 1 − H ^ ​ ( t ) ) (1-\hat{H}(t)) is the normalized certainty. The key design choice is using unnormalized − log ⁡ p -\log p to anchor the ranking: this ensures locally correct tokens (typically high-probability) are naturally excluded, while genuinely anomalous outputs — where the model confidently produced unlikely content — are prioritized. We construct a binary mask by selecting the top-50% of tokens by score: m ( t ) = 𝟏 [ s ( t ) ≥ median ( { s ( t ′ ) } t ′ = 1 T k ) ] . m(t)=\mathbf{1}\left[s(t)\geq\text{median}(\{s(t^{\prime})\}_{t^{\prime}=1}^{T_{k}})\right]. (7)

### 3.5 Unified Objective

The final TTPO objective combines both branches, and balances their weight by λ \lambda : ℒ TTPO = 1 | ℬ | ​ ( ∑ k ∈ 𝒫 ℒ OPSD ​ ( k ) + λ ​ ∑ k ∈ 𝒩 ℒ GRPO ​ ( k ) ) , \mathcal{L}_{\text{TTPO}}=\frac{1}{|\mathcal{B}|}\left(\sum_{k\in\mathcal{P}}\mathcal{L}_{\text{OPSD}}(k)+\lambda\sum_{k\in\mathcal{N}}\mathcal{L}_{\text{GRPO}}(k)\right), (8) The complete training procedure is summarized in Algorithm 1 .

## 4 Experiments

### 4.1 Experimental Setup

#### Implementation.

We evaluate on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B ( Yang et al., 2025 ) , all fine-tuned with LoRA ( r = 64 r{=}64 , α = 128 \alpha{=}128 ) on all linear layers. We consider two settings: (1) OpenThoughts setting , where models are trained on labeled data but TTPO does not use the labels — they serve only for comparison with label-dependent baselines; and (2) TTT setting , where models are trained directly on the test set without any annotations. The shared training configuration follows OPSD, and full training configurations are provided in Appendix A .

#### Baselines.

We compare against: (1) OPSD ( Zhao et al., 2026 ) , on-policy self-distillation with ground-truth labels; (2) GRPO ( Shao et al., 2024 ) , RL with ground-truth rewards; (3) TTRL ( Zuo et al., 2026 ) , label-free RL via majority-vote rewards; and (4) OPSD-TTT , self-distillation using the model’s temperature-0 output under thinking mode as privileged information.

#### Evaluation.

We evaluate on five competition-level math benchmarks: AIME 2025, AIME 2026, HMMT 2025, HMMT 2026, and BRUMO 2025 ( Dekoninck et al., 2026 ) . To ensure train-inference consistency, evaluation is performed with thinking mode enabled (non-thinking evaluation in Appendix D.2 ). All results are reported as Avg@12 with temperature 1.0. For OPSD and TTPO, we train for 100 steps and report the peak performance across checkpoints saved every 25 steps. For GRPO and TTRL, we train for 500 steps and report the peak across all checkpoints.

#### TTPO-specific hyperparameters.

We sample K = 64 K{=}64 trajectories per problem to ensure reliable majority voting on hard problems with low pass rates, with a maximum generation length of 16,000 tokens to avoid truncation that prevents answer extraction. From the K K rollouts, K train = 8 K_{\text{train}}{=}8 are selected (50% positive, 50% negative) for the gradient update, and the RL weight λ = 0.1 \lambda{=}0.1 balances gradient magnitudes between the two branches (ablated in Appendix D.3 ).

### 4.2 Main Results

#### Labeled training data.

Table 1 compares methods trained on OpenThoughts, where OPSD and GRPO use ground-truth labels while TTPO relies solely on majority-vote pseudo-labels. TTPO exceeds the label-dependent OPSD across all three model scales (40.1 vs. 39.7 on 1.7B, 58.6 vs. 58.4 on 4B, 62.6 vs. 61.7 on 8B in average), despite without ground-truth supervision. This demonstrates that majority-vote pseudo-labels, when combined with our asymmetric objective, can substitute for ground-truth annotations without sacrificing performance. The improvements are consistent across scales, and notably, TTPO on Qwen3-4B (58.6 avg) already matches the Qwen3-8B base model (58.6 avg), suggesting that our training recipe effectively amplifies a smaller model’s reasoning capacity to the level of a 2 × 2\times larger untrained model.

#### Label-free test-time training.

Table 2 evaluates the purely label-free TTT setting where models train directly on the test problems. TTPO consistently and substantially outperforms both TTRL and OPSD-TTT across all model scales. On Qwen3-1.7B, TTPO achieves 45.2 average — +3.3 over OPSD-TTT and +5.4 over TTRL, representing a 7.2-point absolute gain over the base model. The gap over TTRL demonstrates the value of dense distributional guidance: while TTRL provides only binary reward signals, TTPO additionally leverages the answer-conditioned teacher to transfer token-level knowledge on correct trajectories. The gap over OPSD-TTT — which uses deterministic (greedy decoding with thinking mode enabled) answers as privileged information rather than majority-vote pseudo-labels — shows that even with a reasonable self-distillation baseline, our asymmetric design extracts substantially more signal by additionally exploiting negative samples through selective RL penalties. Cross-scale comparison further highlights the efficiency: TTPO on Qwen3-4B (61.1 avg) already surpasses Qwen3-8B base (60.7 avg), demonstrating that label-free test-time training with TTPO can close the gap between model sizes.

### 4.3 Ablation Studies

#### Token-level selection.

Table 3 isolates the contribution of each token-level selection mechanism. Both components improve over uniform updates, but their effects are complementary and target different failure modes: removing positive-sample weighting (w/o pos. weight) uniformly distills all tokens including low-value (low-entropy, low-divergence) positions where the student has already converged, diluting the gradient signal from genuinely informative tokens; removing negative-sample masking (w/o neg. mask) penalizes all tokens indiscriminately — not only causing collateral damage to locally correct reasoning steps that cannot be offset without positive-advantage updates, but also allowing anomalous (low-probability, low-entropy) tokens to dominate gradient updates, injecting substantial noise into optimization. The full method benefits from both — focusing distillation where it matters and penalizing only where errors originate.

#### Update strategy.

Figure 3 compares update strategy combinations. The full TTPO (pos=FKL, neg=GRPO, 48.9) substantially outperforms all alternatives. FKL is well-suited to positive samples because their answers match the pseudo-label by definition: even when the label is wrong, the teacher is conditioned on the same answer the student produced, reducing to thinking-to-non-thinking distillation (45.7) that safely transfers careful reasoning. Hence positive-only FKL (46.7) outperforms all-FKL (46.3) and negative-only FKL (43.9), which forces the teacher to steer unmatched answers and injects corrupted signals. GRPO is better suited to negative samples: although its credit assignment is coarser, on hard TTT data most negatives are correctly identified (answer ≠ \neq pseudo-label ∧ \land ≠ \neq ground truth), so a label-free penalty is strictly safer than corrupted distillation. GRPO on positives (37.2) lacks this robustness—it directly reinforces trajectories, and wrong pseudo-labels reverse the update with no mitigation. The reversed assignment (pos=GRPO, neg=FKL, 37.2) therefore performs worst, combining brittle reinforcement on positives with corrupted distillation on negatives.

### 4.4 Analysis

#### Privileged information.

Table 4 ablates privileged information under different teacher modes. With a thinking-mode teacher, the teacher–student distributional gap is already large, so a short answer suffices as a lightweight hint that steers the teacher without crowding out its reasoning. Even a wrong pseudo-label remains consistent with the student’s answer in positive samples, and the update degenerates into thinking-to-non-thinking distillation—still a well-posed and beneficial signal (46.5 vs. 45.8). A full trajectory, by contrast, dominates the context and reduces the teacher to completing a given prefix rather than reasoning independently, degrading performance (41.1). With a non-thinking teacher, the gap is inherently small: a short answer barely shifts the distribution (33.6), while a full trajectory supplies needed context (40.8) but leaves both sides under weak reasoning, making training highly sensitive to pseudo-label noise. The thinking-teacher + answer setting thus offers the best trade-off between guidance and robustness.

#### Generalization beyond the target task.

To verify that TTPO acquires generalizable reasoning improvements rather than overfitting to specific problems, we train on each benchmark separately and evaluate on all three (Figure 4 ). Models trained on any single benchmark consistently improve on the other two as well. This cross-benchmark transfer confirms that TTPO strengthens underlying reasoning capabilities rather than memorizing problem-specific patterns.

#### Upper bound with labeled supervision.

We replace majority-vote pseudo-labels with ground-truth answers to probe the performance ceiling (Figure 5 ). Surprisingly, TTPO with pseudo-labels outperforms both TTPO w/ GT and OPSD (Leakage). First, perfectly correct labels are hard to match on difficult problems, yielding few or zero positives per instance; this starves the FKL branch and leaves GRPO with near-zero advantages that barely penalize negatives (Figure 8 ). Majority-vote labels, being easier to match, keep a healthy positive–negative split and both branches active. Second, AIME26 ground-truth answers are short numbers that barely shift the thinking teacher, unlike the richer OpenThoughts trajectories—reliable, yet too brief to guide strongly. The entropy plot (right) corroborates this: TTPO with pseudo-labels sustains higher entropy, as majority voting and distillation jointly promote exploration that compensates for—and ultimately surpasses—the theoretical benefit of perfect labels.

#### Sustainable self-evolution.

Since majority voting generates the training signal, the base model’s Maj@12 sets the initial ceiling on pseudo-label quality. We track Avg@12 and Maj@12 throughout training to examine whether TTPO can break this ceiling (Figure 6 ). As training progresses, Avg@12 rises steadily to the base Maj@12, confirming that the collective knowledge in majority voting is distilled into single-sample performance. More importantly, Maj@12 does not stagnate but rises in tandem: as the model improves, higher-quality rollouts yield more accurate pseudo-labels, which in turn raise the training ceiling for later steps. This self-evolving cycle enables TTPO to improve beyond its initial supervision and ultimately outperform training with ground-truth that exceed the model’s current capacity (Figure 5 , left).

## 5 Conclusion

We introduced TTPO, which brings OPSD into label-free test-time training by combining it with RL under an asymmetric objective. Distillation provides dense token-level guidance on positives, while RL supplies a robust, label-free penalty on negatives. With token-level selection on both branches, TTPO matches ground-truth-supervised methods, substantially outperforms existing label-free approaches, and exhibits self-evolution with strong cross-task generalization.

## Appendix A Implementation Details

We provide complete training and evaluation configurations in Tables 5 and 6 . All experiments use the AdamW ( Loshchilov & Hutter, 2017 ) optimizer with bfloat16 precision and Flash Attention 2. TTPO experiments use 4 × \times H20 GPUs; all other methods use 8 × \times H20 GPUs. We adopt full-vocabulary logit distillation for all distillation-based methods (OPSD, OPSD-TTT, and TTPO). Following Zhao et al. (2026) , we use a thinking-mode-off student / thinking-mode-on teacher configuration, and the teacher is fixed to base model weights (LoRA adapters disabled) throughout training. For TTPO and OPSD-TTT, we set the maximum sampling length to 16,000 tokens to reduce answer extraction failures caused by truncation during majority voting, while the gradient update still only applies to the first 1,024 completion tokens.

## Appendix B Prompt Templates

We list the prompt templates used for both the student and teacher. All prompts are wrapped with the model’s chat template via apply_chat_template . When privilege_info = none (the no-privilege ablation in Table 4 ), the teacher prompt is identical to the student prompt but with thinking mode enabled.

## Appendix C Case Study: Token-Level Selection Visualization

We visualize how our token weighting (§ 3.3 ) and token masking (§ 3.4 ) operate on a concrete example. Excerpts are drawn from a positive sample and a negative sample generated for the same geometry problem. 1 1 1 Problem: Given square A ​ B ​ C ​ D ABCD with L L on C ​ D CD , K K on extension of D ​ A DA beyond A A , ∠ ​ K ​ B ​ L = 90 ∘ \angle KBL=90^{\circ} , K ​ D = 19 KD=19 , C ​ L = 6 CL=6 . Find L ​ D LD . Ground truth: 7 7 . The positive sample reaches the correct answer 7 7 ; the negative arrives at 397 \sqrt{397} . Recall that token weighting down-weights low-entropy, low-divergence positions where the student has converged, while token masking suppresses low-probability, low-entropy positions representing anomalous model outputs.

### C.1 Token Weighting

The contrast is stark: coordinate values (“ ( 0 , a ) (0,a) ”, “ ( 0 , 0 ) (0,0) ”, “ x = 0 x=0 ”) receive near-zero weight—these are deterministic once the setup is chosen, and both student and teacher assign near-unit probability to each digit. High weight concentrates on geometric insights that determine the solution strategy: “the extension beyond A A is the line… going downwards” (identifying the geometric locus) and “So K K is at ( 0 , y ) (0,y) , for some y < 0 y<0 ” (drawing the conclusion). These are positions where the student is uncertain which geometric fact to invoke or the teacher favors a different continuation, making them the sole source of learning signal. Token weighting thus focuses distillation on where to reason and what to conclude , not on reproducing mechanical substitutions the model already handles reliably.

### C.2 Token Masking

The kept tokens are precisely the positions worth penalizing: local errors (“which is vertical”— C ​ D CD is actually horizontal under the model’s own coordinates), context-inconsistent expressions (“we move… units from C C along C ​ D CD ”—applying a vertical displacement to a horizontal segment), and the resulting anomalous conclusion (“So, L = L= ”—committing to coordinates that place L L off the intended side). Masked-out tokens, by contrast, are locally correct arithmetic (“ ( s , s ) (s,s) ”, “ = 6 =6 ”) and formatting that would appear identically in a correct solution; penalizing them would damage valid computation skills without addressing the actual reasoning flaw. Token masking thus restricts the penalty gradient to the confident errors that cause failure —wrong geometric claims and their immediate consequences—while leaving shared, reusable sub-skills intact.

### C.3 Complementarity

The two mechanisms implement a dual philosophy: token weighting asks “ where does the model still need to learn? ” and suppresses already-converged positions in positive samples; token masking asks “ where is the model confidently wrong? ” and suppresses locally-correct or uncertain positions in negative samples. Both avoid wasting gradient on low-signal tokens—from opposite directions—yielding more efficient and stable training.

## Appendix D Additional Experiments

### D.1 Training Dynamics

Figure 7 compares training dynamics of Qwen3-1.7B TTT on AIME 2026 across methods. The most striking observation is that TTPO w/ GT exhibits dramatically weaker training signal than all other methods: its loss barely decreases and frequently stagnates near zero. This directly validates our theoretical analysis (Eq. 13 – 14 ) — on hard AIME problems where | 𝒫 GT | ≈ 0 |\mathcal{P}_{\text{GT}}|\approx 0 , GT routing starves both branches simultaneously. In contrast, TTPO with majority-vote pseudo-labels maintains a steady and substantial loss decrease throughout training, confirming that vote-based routing keeps both branches active. OPSD (Leakage) and OPSD-TTT both show consistent loss reduction; notably, TTPO achieves comparable or stronger training dynamics despite operating in a fully label-free setting.

### D.2 Non-Thinking Evaluation

Table 7 evaluates models with thinking mode disabled to assess whether training with a thinking-mode teacher transfers reasoning capabilities to non-thinking inference. TTPO achieves dramatically larger gains than OPSD across all scales: on average, TTPO improves over the base model by +25.2 (1.7B), +30.6 (4B), and +36.4 (8B) points, while OPSD improves by only +7.1, +5.8, and +3.5 points respectively. This indicates that TTPO far more effectively absorbs the thinking teacher’s reasoning ability into the student’s non-thinking distribution. We attribute this to the asymmetric objective: the GRPO branch on negative samples directly penalizes poor reasoning patterns in the student’s own generation mode, while OPSD’s pure distillation only passively aligns the student toward the teacher without actively suppressing failure modes.

### D.3 Additional Ablations

#### RL weight λ \lambda .

As shown in Figure 8 , the raw GRPO loss is roughly an order of magnitude larger than the OPSD forward-KL loss, while the positive sample fraction increases over training as the model produces more correct answers. Without scaling, the GRPO branch would dominate gradients and destabilize training. We therefore introduce a weight λ \lambda on the GRPO loss to balance the two branches. Table 8 sweeps λ ∈ { 0.01 , 0.05 , 0.1 , 0.15 , 0.2 } \lambda\in\{0.01,0.05,0.1,0.15,0.2\} : performance peaks at λ = 0.1 \lambda{=}0.1 , which approximately equalizes the gradient magnitudes of the two branches. Both under-weighting ( λ ≤ 0.05 \lambda{\leq}0.05 , insufficient negative penalty) and over-weighting ( λ ≥ 0.15 \lambda{\geq}0.15 , excessive penalty dominating distillation) degrade results.

#### K train K_{\text{train}} positive-negative fraction.

Table 10 ablates the positive-negative composition of the K train K_{\text{train}} subset. A fixed 50/50 split outperforms both random sampling and a dynamic fraction. The dynamic strategy faces a fundamental dilemma: when the positive fraction in K K is high (i.e., pseudo-label is likely correct), proportionally reducing negative samples in K train K_{\text{train}} cancels the amplified group-relative advantages that negative samples receive — the enlarged signal is immediately diluted by fewer recipients. Conversely, if the dynamic strategy inverts the ratio (more negatives when positives dominate), it sacrifices the reliable fine-grained FKL supervision available precisely when the pseudo-label is most trustworthy, replacing it with coarser GRPO penalties. Either direction has drawbacks; a fixed 50/50 split avoids both failure modes and guarantees stable gradient contributions from both branches at every step.

#### K train K_{\text{train}} selection strategy.

Table 10 compares strategies for selecting which rollouts enter K train K_{\text{train}} . Selecting the shortest completions performs best. Since only the first 1,024 tokens participate in the gradient update, shorter trajectories ensure that these tokens constitute a larger fraction of the total reasoning chain and are more likely to contain the critical steps that determine the final answer. For longer trajectories, the first 1,024 tokens often cover only preliminary exploration, with the decisive reasoning occurring well beyond the gradient window — yielding little useful learning signal. The “Top signal” strategy selects positive samples with the highest teacher-student FKL divergence (intuitively, trajectories where the student deviates most from the teacher and thus has the most to learn) and negative samples with the highest log-probability (intuitively, confident errors that carry the strongest penalty signal). Despite this seemingly stronger per-sample signal, the strategy underperforms shortest selection — the intuition that larger divergence or higher confidence implies more useful gradients lacks theoretical grounding and appears unreliable in practice.

## Appendix E Theoretical Analysis

### E.1 Setup

For problem x x with ground-truth a ∗ a^{*} and pseudo-label a ^ \hat{a} : q t ( a ) = π θ ( ⋅ ∣ [ x ; a ] , y < t ) , p t = π θ ( ⋅ ∣ x , y < t ) , ∇ θ KL ( q t ( a ) ∥ p t ) = − ∑ v q t ( a ) ( v ) ∇ θ log p t ( v ) . \displaystyle q_{t}^{(a)}=\pi_{\theta}(\cdot\mid[x;a],y_{<t}),\quad p_{t}=\pi_{\theta}(\cdot\mid x,y_{<t}),\quad\nabla_{\theta}\mathrm{KL}(q_{t}^{(a)}\|p_{t})=-\sum_{v}q_{t}^{(a)}(v)\,\nabla_{\theta}\log p_{t}(v). (9)

### E.2 FKL Signal Analysis

#### Positive samples ( y k ∈ 𝒫 y_{k}\in\mathcal{P} , a k = a ^ a_{k}=\hat{a} ).

The trajectory agrees with teacher, so the FKL collapses to: KL ( q t ( a ^ ) ∥ p t ) | a k = a ^ = KL ( q t ∥ p t ) ⏟ thinking vs. non-thinking only . \mathrm{KL}\!\left(q_{t}^{(\hat{a})}\,\big\|\,p_{t}\right)\bigg|_{a_{k}=\hat{a}}\;=\;\underbrace{\mathrm{KL}(q_{t}\|p_{t})}_{\text{thinking vs.\ non-thinking only}}. (10) If a ^ = a ∗ \hat{a}=a^{*} this recovers standard OPSD; if a ^ ≠ a ∗ \hat{a}\neq a^{*} it reduces to answer-agnostic distillation.

#### Negative samples ( y k ∈ 𝒩 y_{k}\in\mathcal{N} , a k ≠ a ^ a_{k}\neq\hat{a} ).

The disagreement introduces a non-negative conflict term: KL ( q t ( a ^ ) ∥ p t ) | a k ≠ a ^ = KL ( q t ∥ p t ) ⏟ thinking vs. non-thinking + Δ conflict ​ ( t ) ⏟ ≥ 0 , \mathrm{KL}\!\left(q_{t}^{(\hat{a})}\,\big\|\,p_{t}\right)\bigg|_{a_{k}\neq\hat{a}}\;=\;\underbrace{\mathrm{KL}(q_{t}\|p_{t})}_{\text{thinking vs.\ non-thinking}}\;+\;\underbrace{\Delta_{\text{conflict}}(t)}_{\geq\,0}, (11) where Δ conflict ​ ( t ) \Delta_{\text{conflict}}(t) captures the teacher’s pressure to redirect reasoning from a k a_{k} toward a ^ \hat{a} . This is beneficial when a ^ = a ∗ \hat{a}=a^{*} , but harmful when a ^ ≠ a ∗ \hat{a}\neq a^{*} , particularly when a k = a ∗ a_{k}=a^{*} , as the gradient actively suppresses correct reasoning.

#### Method comparison.

By Eq. 10 , FKL on 𝒫 \mathcal{P} is at worst benign and at best recovers standard OPSD; GRPO on 𝒩 \mathcal{N} is label-agnostic. U-OPSD applies FKL to 𝒩 \mathcal{N} instead, where Δ conflict \Delta_{\text{conflict}} (Eq. 11 ) misdirects correct trajectories whenever a ^ ≠ a ∗ \hat{a}\neq a^{*} , and requires reliable pseudo-labels since low consensus makes Δ conflict \Delta_{\text{conflict}} harmful while unanimous consensus yields waste of rollouts ( | 𝒩 | = 0 |\mathcal{N}|=0 ).

### E.3 Majority-Vote vs. Ground-Truth Routing

With binary reward r k = 𝟏 [ a k ∈ majority ] r_{k}=\mathbf{1}[a_{k}\in\text{majority}] , r ¯ = | 𝒫 | / K \bar{r}=|\mathcal{P}|/K . The GRPO advantage for k ∈ 𝒩 k\in\mathcal{N} : A k = − | 𝒫 | / K 1 − | 𝒫 | / K , k ∈ 𝒩 . A_{k}=-\sqrt{\frac{|\mathcal{P}|/K}{1-|\mathcal{P}|/K}},\quad k\in\mathcal{N}. (12)

#### GT routing

( 𝒫 GT = { k : a k = a ∗ } \mathcal{P}_{\text{GT}}=\{k:a_{k}=a^{*}\} ). On hard problems with small | 𝒫 GT | |\mathcal{P}_{\text{GT}}| , both branches vanish: | 𝒫 GT | K ≈ 0 \displaystyle\frac{|\mathcal{P}_{\text{GT}}|}{K}\approx 0\quad and ∑ k ∈ 𝒫 GT ∇ θ KL ( q t ∥ p t ) ⏟ FKL ≈ 𝟎 , \displaystyle\text{and}\quad\underbrace{\sum_{k\in\mathcal{P}_{\text{GT}}}\nabla_{\theta}\mathrm{KL}(q_{t}\|p_{t})}_{\text{FKL}}\approx\mathbf{0}, (13) | A k | = | 𝒫 GT | / K 1 − | 𝒫 GT | / K ≈ 0 \displaystyle|A_{k}|=\sqrt{\frac{|\mathcal{P}_{\text{GT}}|/K}{1-|\mathcal{P}_{\text{GT}}|/K}}\approx 0\quad and A k ​ ∇ θ ​ log ⁡ π θ ​ ( y k ∣ x ) ⏟ GRPO ≈ 𝟎 . \displaystyle\text{and}\quad\underbrace{A_{k}\,\nabla_{\theta}\log\pi_{\theta}(y_{k}\mid x)}_{\text{GRPO}}\approx\mathbf{0}. (14)

#### Vote routing

( 𝒫 vote = { k : a k = a ^ } \mathcal{P}_{\text{vote}}=\{k:a_{k}=\hat{a}\} ). The model always forms a non-empty consensus | 𝒫 vote | > 0 |\mathcal{P}_{\text{vote}}|>0 , so both branches remain active: | 𝒫 vote | K > 0 \displaystyle\frac{|\mathcal{P}_{\text{vote}}|}{K}>0\quad and ∑ k ∈ 𝒫 vote ∇ θ KL ( q t ∥ p t ) ⏟ FKL ≠ 𝟎 , \displaystyle\text{and}\quad\underbrace{\sum_{k\in\mathcal{P}_{\text{vote}}}\nabla_{\theta}\mathrm{KL}(q_{t}\|p_{t})}_{\text{FKL}}\neq\mathbf{0}, (15) A k = − | 𝒫 vote | / K 1 − | 𝒫 vote | / K < 0 \displaystyle A_{k}=-\sqrt{\frac{|\mathcal{P}_{\text{vote}}|/K}{1-|\mathcal{P}_{\text{vote}}|/K}}<0\quad and A k ​ ∇ θ ​ log ⁡ π θ ​ ( y k ∣ x ) ⏟ GRPO ≠ 𝟎 . \displaystyle\text{and}\quad\underbrace{A_{k}\,\nabla_{\theta}\log\pi_{\theta}(y_{k}\mid x)}_{\text{GRPO}}\neq\mathbf{0}. (16) As the model improves, a ^ \hat{a} converges toward a ∗ a^{*} (Figure 6 ).

## Appendix F Limitations and Future Work

#### Dependence on majority-vote quality.

TTPO relies on majority voting to generate pseudo-labels and classify positive/negative samples. When the sample budget K K is very small or the problem is so difficult that no rollout produces a correct answer, the voting signal degrades and both branches receive noisy supervision. Adaptive strategies that adjust the positive-negative ratio or fall back to pure RL under low-consensus conditions could mitigate this.

#### Domain scope.

Our experiments are restricted to mathematical reasoning with verifiable final answers. Extending TTPO to domains where correctness is harder to extract automatically — such as code generation (requiring execution-based verification) or open-ended reasoning (requiring learned reward models) — remains unexplored.

#### Dynamic training curriculum.

TTPO applies a fixed asymmetric objective throughout training. As the model improves and pseudo-label accuracy rises, the optimal balance between distillation and RL may shift. A curriculum that dynamically adjusts the RL weight or positive-negative fraction to training dynamics could further improve efficiency.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .