TTRL: Test-Time Reinforcement Learning

arxiv:2504.16084

Original source · Paper JSON

Post-training · 2025-04-22

Cites (9) · Cited by (15)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• TTRL: Test-Time Reinforcement Learning
• Abstract
• 1 Introduction
• 2 Test-Time Reinforcement Learning (TTRL)
• 2.1 Methodology
• 2.2 Majority Voting Reward Function
• 3 Experiments
• 3.1 Experimental Setup
• Models
• Benchmarks
• Evaluation Setup
• Baselines
• Implementation Details
• 3.2 Main Results
• TTRL performs well on most tasks and models.
• TTRL performs well on LRMs.
• TTRL naturally scales.
• TTRL generalizes well beyond the target task.
• TTRL is compatible with different RL algorithms.
• TTRL achieves sustainable self-evolution through “online” and “RL”.
• 4 Analysis and Discussions
• 4.1 Q1: How Well Can TTRL Perform?
• 4.2 Q2: Why Does TTRL Work?
• Label Estimations.
• Reward Calculations.
• Online Learning.
• 4.3 Q3: When Might TTRL Fail?
• Inappropriate RL Hyperparameters.
• Lack of Prior Knowledge on Target Task.
• 5 Related Works
• 5.1 Test-Time Scaling
• 5.2 RL for Reasoning
• 6 Conclusion
• 7 Limitations and Future Works
• Limitations
• Future Works
• Appendix A Additional Results
• Appendix B Training Metrics
• Appendix C Terminology
• C.1 Test-Time Training (TTT)
• C.2 Test-Time Inference (TTI)
• Instructions for reporting errors