TTPO: Test-Time Policy Optimization — part 1/3
Cleaned-text code points [0, 15993). Bibliography omitted here; source.md includes it.
# TTPO: Test-Time Policy Optimization
###### Abstract
Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO) , an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization. Our code is available at https://github.com/ZJU-REAL/TTPO .
## 1 Introduction
Large language models (LLMs) have achieved remarkable mathematical reasoning through extended chain-of-thought generation ( Team et al., 2026 ; Xu et al., 2026a ; Zeng et al., 2026 ; Singh et al., 2025 ; Team, 2026 ) , largely powered by post-training with reinforcement learning from verifiable rewards (RLVR) ( Shao et al., 2024 ; Yu et al., 2026 ; Guo et al., 2025 ) . Yet such outcome rewards are inherently coarse, broadcasting a single sequence-level scalar uniformly across all tokens and leaving the reasoning at each step unsupervised.
A complementary line supplies dense, token-level supervision: on-policy self-distillation (OPSD) ( Zhao et al., 2026 ) conditions the same policy on the ground-truth answer to form a teacher that re-scores the student’s own rollouts token by token ( Ye et al., 2026 ; Yang et al., 2026b ) . Recent work increasingly combines the two signals, through auxiliary distillation losses, credit redistribution, or routing between them ( Yang et al., 2026a ; Liu et al., 2026b ; Lu et al., 2026a ; Han et al., 2026 ; Li et al., 2026a ) . All these methods, however, assume ground-truth answers: the reward needs them for verification, and the teacher needs them as privileged context. In test-time training (TTT) ( Sun et al., 2020 ) , where a model improves on the very problems it must solve and labels never arrive, none of these methods applies.
Without labels, supervision must come from the model itself: sample a group of rollouts for each problem, and take the majority answer as a pseudo-label. TTRL ( Zuo et al., 2026 ) uses this pseudo-label as a reward and improves reasoning without any labels, but the reward is still one scalar per trajectory, and when the majority is wrong, training reinforces the error ( Lin et al., 2026 ) . The natural next step is to let the pseudo-label replace the ground-truth answer in OPSD, which has been tried in two forms: distilling all rollouts toward the pseudo-label-conditioned teacher ( Gkountouras et al., 2026 ) , or distilling only the rollouts that disagree with the pseudo-label ( Li et al., 2026b ) . But dense supervision magnifies label errors: a corrupted reward misleads once per trajectory; a corrupted teacher misleads at every token .
These errors are the common case: on competition-level problems, the pseudo-label is wrong for ∼ {\sim} 85% of prompts (Figure 1 , a). Learning from such a label seems infeasible. However, even when the pseudo-label is wrong, ∼ {\sim} 79% of the rollouts that disagree with it are wrong too. A penalty on a disagreeing rollout is therefore usually correct whether or not the pseudo-label is, because it uses the disagreement alone, never the pseudo-label’s answer. Distillation toward the pseudo-label has no such tolerance: a wrong answer enters the teacher and misleads every token. The same asymmetry underlies negative learning from noisy labels, where stating what a sample is not remains reliable even when the label is wrong ( Kim et al., 2019 ) .
We propose Test-Time Policy Optimization (TTPO), which applies each signal where it is reliable: GRPO penalties on the rollouts that disagree with the pseudo-label, and OPSD distillation on the rollouts that agree with it. The distillation branch tolerates wrong pseudo-labels for a different reason: the teacher is conditioned on the answer that the agreeing rollouts themselves produced, so even when that answer is wrong, the update distills the model’s thinking mode into its non-thinking mode rather than toward an arbitrary error. Finally, token-level selection sharpens both branches, weighting distillation toward positions the student has not yet mastered and masking penalties to the confident errors that caused the failure. By combining both signals, our asymmetric design not only makes more effective use of both positive and negative rollouts while remaining robust to pseudo-label noise, but is also naturally calibrated to the model’s current capability, enabling a virtuous cycle of self-evolution that ground-truth routing cannot sustain (Figure 5 , 6 ).
Trained without any labels, TTPO matches or exceeds label-supervised OPSD across Qwen3-1.7B/4B/8B on five competition-level benchmarks, and in the pure TTT setting raises the 1.7B base model from 38.0% to 45.2% average accuracy, ahead of both TTRL and self-distillation baselines. With thinking mode disabled, the gains reach +25.2% to +36.4% across scales, several times the gain of label-supervised OPSD. We further validate that training on any one benchmark improves the other two, indicating generalizable reasoning rather than problem-specific overfitting (Figure 4 ). Our contributions are: 1. We show that majority-vote pseudo-labels remain useful despite frequent errors: though wrong on ∼ {\sim} 85% of competition-level prompts, ∼ {\sim} 79% of disagreeing rollouts are wrong too, so penalizing disagreement stays correct while distillation does not.
2. We propose TTPO, which applies each signal where it stays correct: agreeing rollouts are distilled toward an answer-conditioned teacher, disagreeing rollouts receive GRPO penalties, with token-level selection in both branches.
3. Trained without any labels, TTPO matches or exceeds label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, and further demonstrates strong cross-task generalization.
## 2 Related Work
### 2.1 Test-Time Training for Reasoning
Test-time training (TTT) adapts models to unlabeled test data at inference time ( Sun et al., 2020 ; Li et al., 2026b ; Du et al., 2025 ) . TTRL ( Zuo et al., 2026 ) extends TTT to LLM reasoning by sampling multiple trajectories per problem, deriving pseudo-rewards via majority voting, and training with GRPO ( Shao et al., 2024 ) . Follow-up work addresses TTRL’s sensitivity to consensus quality: Hi-TTRL ( Xu et al., 2026b ) introduces hierarchical reward shaping with hints, while SCRL ( Yan et al., 2026 ) applies selective pseudo-labeling to filter unreliable majorities. However, these methods remain purely RL-based, propagating a single sequence-level reward uniformly across all tokens.
### 2.2 On-Policy Self-Distillation
On-policy distillation trains a policy on its own rollouts under a teacher ( Agarwal et al., 2024 ; Gu et al., 2026 ; Wen et al., 2023 ) . Recent self-distillation variants remove the need for a separate teacher by conditioning the same model on privileged information available only during training ( Zhao et al., 2026 ; He et al., 2026 ; Lu et al., 2026b ) . Several studies further incorporate the resulting teacher–student log-probability gap into RLVR, either as advantage scaling ( Yang et al., 2026a ) , a detached auxiliary objective ( Lu et al., 2026a ) , a routing mechanism ( Han et al., 2026 ; Li et al., 2026a ) , or reward-densifying local supervision ( Xu et al., 2026c ; Ye et al., 2026 ; He et al., 2026 ) . We follow this idea and introduce an asymmetric objective that decouples the treatment of positive and negative samples to tolerate pseudo-label errors.
### 2.3 Token-Level Weighting and Masking
Recent work recognizes that not all tokens merit equal gradients during training ( Xiao et al., 2026 ) . In distillation, TIP ( Xu et al., 2026c ) shows that training on fewer than 10% of tokens, selected by student entropy and teacher–student divergence, nearly matches full-token performance. In RL, STAPO ( Liu et al., 2026a ) masks spurious low-probability, low-entropy tokens in positive samples that receive disproportionate reward gradients, while Wu et al. (2026) address the erroneous penalization of locally correct tokens within failed trajectories through reward recalibration. TTPO applies token-level selection to both branches of its objective: down-weighting converged positions in the distillation branch and selectively penalizing only confident errors in the RL branch.
## 3 Method
### 3.1 Preliminaries and Problem Setup
Let π θ \pi_{\theta} denote the language model and { x i } i = 1 N \{x_{i}\}_{i=1}^{N} be a set of test-time problems without ground-truth labels. For each problem x x , we sample K K trajectories { y 1 , … , y K } ∼ π θ ( ⋅ ∣ x ) \{y_{1},\ldots,y_{K}\}\sim\pi_{\theta}(\cdot\mid x) and extract final answers a k = Extract ( y k ) a_{k}=\operatorname{Extract}(y_{k}) .
#### Majority-vote pseudo-labeling.
We cluster answers by mathematical equivalence, select the largest cluster as the pseudo-label a ^ \hat{a} with consensus count c = | { k : a k ≡ a ^ } | c=|\{k:a_{k}\equiv\hat{a}\}| , and partition trajectories into positive samples 𝒫 = { k : a k ≡ a ^ } \mathcal{P}=\{k:a_{k}\equiv\hat{a}\} that agree with pseudo-label and negative samples 𝒩 = { k : a k ≢ a ^ } \mathcal{N}=\{k:a_{k}\not\equiv\hat{a}\} that disagree.
#### Answer-conditioned teacher.
Following OPSD ( Zhao et al., 2026 ) , we construct a teacher by conditioning the same model on the pseudo-label a ^ \hat{a} as privileged information. Given shared completion tokens y k y_{k} , the teacher and student differ only in their prompt prefixes: q t ( a ^ ) \displaystyle q_{t}^{(\hat{a})} = π θ ( ⋅ ∣ [ x ; a ^ ] teacher , y < t ) , (no grad) \displaystyle=\pi_{\theta}(\cdot\mid[x;\hat{a}]_{\text{teacher}},y_{<t}),\quad\text{(no grad)} (1) p t \displaystyle p_{t} = π θ ( ⋅ ∣ x student , y < t ) . (with grad) \displaystyle=\pi_{\theta}(\cdot\mid x_{\text{student}},y_{<t}).\quad\text{(with grad)} (2)
### 3.2 Motivation: Why Asymmetric?
In the TTT setting, the training set is the test set — consisting of competition-level problems that are inherently difficult for the model. As a result, majority-vote pseudo-labels are frequently wrong: on AIME 2026 with Qwen3-1.7B, we observe that pseudo-labels are wrong for ∼ {\sim} 85% of prompts on average (Figure 1 , a). Naively applying self-distillation to all trajectories using the corrupted teacher would propagate errors to every sample.
However, we observe a key structural property: even when a ^ ≠ a ∗ \hat{a}\neq a^{*} , ∼ {\sim} 79% of negative samples produce answers that are neither a ^ \hat{a} nor a ∗ a^{*} — penalizing them is correct regardless of pseudo-label quality. This motivates an asymmetric design that minimizes the blast radius of pseudo-label errors: • Pure FKL on all samples : error propagates to all K K trajectories (the teacher distribution is entirely conditioned on wrong a ^ \hat{a} ).
• Asymmetric (FKL for 𝒫 \mathcal{P} , GRPO for 𝒩 \mathcal{N} ) : only the small | 𝒫 | |\mathcal{P}| set is corrupted. GRPO on 𝒩 \mathcal{N} requires only “not in the majority cluster” — independent of a ^ \hat{a} ’s content — and is correct for the vast majority of negative samples.
### 3.3 Positive Samples: OPSD Branch
For each positive sample y k ∈ 𝒫 y_{k}\in\mathcal{P} , we apply OPSD’s forward KL with per-token weighting: ℒ OPSD ( k ) = 1 T k ∑ t = 1 T k w ( t ) ⋅ KL ( q t ( a ^ ) ∥ p t ) , \mathcal{L}_{\text{OPSD}}(k)=\frac{1}{T_{k}}\sum_{t=1}^{T_{k}}w(t)\cdot\text{KL}\left(q_{t}^{(\hat{a})}\,\|\,p_{t}\right), (3) where T k T_{k} is the number of valid response tokens and w ( t ) w(t) is the token weight described below.
#### Token weighting.
Not all tokens offer equal learning value. Inspired by the token importance analysis of Xu et al. (2026c) , we design a weighting scheme that down-weights positions where the student has already converged. We measure two complementary signals — student entropy H ( t ) H(t) and teacher-student divergence Δ ( t ) = KL ( q t ∥ p t ) \Delta(t)=\text{KL}(q_{t}\|p_{t}) — normalize each to [ 0 , 1 ] [0,1] via per-sample min-max normalization, and combine them with a Soft-OR: w ( t ) = H ^ ( t ) + Δ ^ ( t ) − H ^ ( t ) ⋅ Δ ^ ( t ) . w(t)=\hat{H}(t)+\hat{\Delta}(t)-\hat{H}(t)\cdot\hat{\Delta}(t). (4) This assigns high weight when at least one signal indicates learning value (the student is uncertain, or confidently wrong), and approaches zero only when both signals are low — the student is already confident and aligned with the teacher.
### 3.4 Negative Samples: GRPO Branch
For negative samples y k ∈ 𝒩 y_{k}\in\mathcal{N} , we apply GRPO ( Shao et al., 2024 ) . Each trajectory receives a binary reward based on majority-vote classification: r k = 1 r_{k}=1 if its answer matches the pseudo-label, r k = 0 r_{k}=0 otherwise. Advantages are computed group-relatively over all K K rollouts per problem, so that A k < 0 A_{k}<0 for negative samples, yielding a penalty: ℒ GRPO ( k ) = − A k T k ∑ t = 1 T k m ( t ) ⋅ log π θ ( y k ( t ) ∣ x , y k ( < t ) ) . \mathcal{L}_{\text{GRPO}}(k)=-\frac{A_{k}}{T_{k}}\sum_{t=1}^{T_{k}}m(t)\cdot\log\pi_{\theta}(y_{k}^{(t)}\mid x,y_{k}^{(<t)}). (5)
In standard GRPO, positive-advantage gradients counterbalance erroneous penalties on locally correct tokens within failed trajectories. In our asymmetric framework, GRPO acts exclusively on negative samples, removing this counterbalance. This False Penalties on Negative Samples problem ( Wu et al., 2026 ) necessitates active reduction of collateral damage.
#### Token masking.
Drawing on the insight from Liu et al. (2026a) that low-probability, low-entropy tokens represent anomalous model behavior, we design a masking scheme to identify tokens most responsible for errors. We score each token by: s ( t ) = − log π θ ( y k ( t ) ∣ x , y k ( < t ) ) ⋅ ( 1 − H ^ ( t ) ) , s(t)=-\log\pi_{\theta}(y_{k}^{(t)}\mid x,y_{k}^{(<t)})\cdot(1-\hat{H}(t)), (6) where the raw negative log-probability serves as the dominant ranking factor and ( 1 − H ^ ( t ) ) (1-\hat{H}(t)) is the normalized certainty. The key design choice is using unnormalized − log p -\log p to anchor the ranking: this ensures locally correct tokens (typically high-probability) are naturally excluded, while genuinely anomalous outputs — where the model confidently produced unlikely content — are prioritized. We construct a binary mask by selecting the top-50% of tokens by score: m ( t ) = 𝟏 [ s ( t ) ≥ median ( { s ( t ′ ) } t ′ = 1 T k ) ] . m(t)=\mathbf{1}\left[s(t)\geq\text{median}(\{s(t^{\prime})\}_{t^{\prime}=1}^{T_{k}})\right]. (7)
### 3.5 Unified Objective
The final TTPO objective combines both branches, and balances their weight by λ \lambda : ℒ TTPO = 1 | ℬ | ( ∑ k ∈ 𝒫 ℒ OPSD ( k ) + λ ∑ k ∈ 𝒩 ℒ GRPO ( k ) ) , \mathcal{L}_{\text{TTPO}}=\frac{1}{|\mathcal{B}|}\left(\sum_{k\in\mathcal{P}}\mathcal{L}_{\text{OPSD}}(k)+\lambda\sum_{k\in\mathcal{N}}\mathcal{L}_{\text{GRPO}}(k)\right), (8) The complete training procedure is summarized in Algorithm 1 .