Weight-Space Geometry of Offline Reasoning Training

arxiv:2606.23740

Original source · Paper JSON

Post-training · 2026-06-26

Cites (4) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• Weight-Space Geometry of Offline Reasoning Training
• Abstract
• Keywords:
• 1 Introduction
• 2 Setup
• Data.
• Methods.
• Training.
• Analysis.
• 3 Results
• 3.1 Downstream accuracy
• On-policy RL preserves accuracy; SFT-style loses it.
• 3.2 Weight-space convergence
• 3.3 Subspace analysis
• Top-1 singular directions.
• 3.4 Seed and learning-rate sensitivity
• 3.5 Linear mode connectivity
• 3.6 Representational similarity
• 4 Discussion
• Reward-weighted MLE is SFT, plus DFT is the surprise.
• Offline GRPO shifts direction but stays in basin; online GRPO does not.
• DPO sits apart, geometrically and on accuracy.
• Limitations.
• Appendix A Supplementary figures
• Instructions for reporting errors