Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision

arxiv:2509.14234

Original source · Paper JSON

Post-training · 2025-09-17

Cites (9) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision
• Abstract
• 1 Introduction
• Compute as Teacher (CaT).
• Reference-free rewards for non-verifiable domains.
• Why it works (intuition).
• Practicality.
• CaT bridges several lines of work.
• Organization.
• 2 Related Work
• Reference-Free Fine-Tuning.
• Reference-Free RL.
• Test-Time Scaling.
• Non-Verifiable RL.
• 3 Compute as Teacher (CaT)
• Notation.
• 3.1 Reference Estimation
• Synthesis.
• When does synthesis help?
• 3.2 Reward Derivation
• 3.2.1 Verifiable Domains
• 3.2.2 Non-Verifiable Domains
• Rubric Generation.
• Rubric Scoring.
• Why Rubrics?
• Example.
• 3.3 Training
• GRPO Overview.
• Integration with CaT.
• Training Loop.
• Inference-Time Use.
• 4 Experiments
• 4.1 Setup
• Datasets.
• Models.
• Evaluation.
• 4.2 Main Results
• CaT matches or exceeds inference-time synthesis.
• CaT enables improvement until rollouts converge.
• 4.3 Self-Proposed Rubrics for Non-Verifiable Domains
• Self-proposed rubrics compete with expert annotations.
• RL with rubrics outperforms SFT.
• 4.4 Reference Estimation: Synthesis vs Selection
• Selection baselines.
• Synthesis matches or exceeds selection methods.
• 4.5 How Does Synthesis Differ From Selection?
• Synthesis disagrees with consensus.
• Synthesis can correct all rollouts.
• Synthesis reasons over multiple rollouts.
• 5 Discussion
• Limitations and future work.
• Conclusion.
• Acknowledgements
• Appendix A Results Tables And Additional Results
• A.1 RL Results Against Selection Methods
• A.2 Conditioning Rubric Generation on the Pseudo-Reference
• A.3 Judge Sensitivity
• A.4 Omitting the Question in the Synthesis Aggregator
• Appendix B Example: Synthesis Disagrees With All Rollouts
• Appendix C Analysis of Synthesis Outputs
• Appendix D When Does CaT Stop Learning?
• Appendix E GRPO Overview
• Appendix F Prompts
• Appendix G Example Rubrics
• Appendix H Hyperparameters
• Appendix I Experimental Details
• Computing perplexity.
• Computing mutual predictability.
• Supervised fine-tuning.
• RL fine-tuning.
• Instructions for reporting errors