TTPO: Test-Time Policy Optimization
arxiv:2608.27448
Post-training · 2026-08-27
Complete cleaned Markdown · Source Markdown including bibliography
Read in parts
Contents
- TTPO: Test-Time Policy Optimization
- Abstract
- 1 Introduction
- 2 Related Work
- 2.1 Test-Time Training for Reasoning
- 2.2 On-Policy Self-Distillation
- 2.3 Token-Level Weighting and Masking
- 3 Method
- 3.1 Preliminaries and Problem Setup
- Majority-vote pseudo-labeling.
- Answer-conditioned teacher.
- 3.2 Motivation: Why Asymmetric?
- 3.3 Positive Samples: OPSD Branch
- Token weighting.
- 3.4 Negative Samples: GRPO Branch
- Token masking.
- 3.5 Unified Objective
- 4 Experiments
- 4.1 Experimental Setup
- Implementation.
- Baselines.
- Evaluation.
- TTPO-specific hyperparameters.
- 4.2 Main Results
- Labeled training data.
- Label-free test-time training.
- 4.3 Ablation Studies
- Token-level selection.
- Update strategy.
- 4.4 Analysis
- Privileged information.
- Generalization beyond the target task.
- Upper bound with labeled supervision.
- Sustainable self-evolution.
- 5 Conclusion
- Appendix A Implementation Details
- Appendix B Prompt Templates
- Appendix C Case Study: Token-Level Selection Visualization
- C.1 Token Weighting
- C.2 Token Masking
- C.3 Complementarity
- Appendix D Additional Experiments
- D.1 Training Dynamics
- D.2 Non-Thinking Evaluation
- D.3 Additional Ablations
- RL weight λ \lambda .
- K train K_{\text{train}} positive-negative fraction.
- K train K_{\text{train}} selection strategy.
- Appendix E Theoretical Analysis
- E.1 Setup
- E.2 FKL Signal Analysis
- Positive samples ( y k ∈ 𝒫 y_{k}\in\mathcal{P} , a k = a ^ a_{k}=\hat{a} ).
- Negative samples ( y k ∈ 𝒩 y_{k}\in\mathcal{N} , a k ≠ a ^ a_{k}\neq\hat{a} ).
- Method comparison.
- E.3 Majority-Vote vs. Ground-Truth Routing
- GT routing
- Vote routing
- Appendix F Limitations and Future Work
- Dependence on majority-vote quality.
- Domain scope.
- Dynamic training curriculum.
- Instructions for reporting errors