Weight-Space Geometry of Offline Reasoning Training — part 2/2

Cleaned-text code points [15840, 20057). Bibliography omitted here; source.md includes it.

CKA on hidden states (Figure 6 , 100 100 GSM8K prompts) confirms the weight-space picture. SFT–RIFT CKA stays above 0.99 0.99 at every layer; SFT/RIFT–Offline GRPO drop to ∼ 0.85 \sim 0.85 in the final layer (GRPO reshapes output-facing layers); Offline GRPO–DPO and DFT–DPO start near 0.93 0.93 and collapse to ∼ 0.45 \sim\!\!0.45 from layer 25 25 onward. A logit-lens probe gives a complementary null — mean prediction depth is 35.3 35.3 – 36.0 36.0 out of 36 36 for every method — but this is partly forced by attention-only LoRA leaving MLPs frozen, so it should not be read as a finding about the losses.

## 4 Discussion

#### Reward-weighted MLE is SFT, plus DFT is the surprise.

SFT, RFT, and RIFT differ only in how they handle negative samples (drop, weight, or include uniformly), yet the resulting LoRA deltas have cosine ≥ 0.97 \geq 0.97 , principal angles < 25 ∘ <25^{\circ} , and indistinguishable per-layer CKA, and they sit within 1 1 percentage point of each other on full GSM8K. The Frobenius norms differ by up to 7 % 7\% . If RIFT outperforms SFT at the same step count, our results suggest the explanation lies in magnitude (effective step size in the SFT direction), not direction — a longer or higher-lr SFT run should close the gap. DFT, by contrast, has cosine ∼ 0.55 \sim 0.55 to SFT/RIFT despite using less information than they do (no reward, no filtering): self-weighting by sg ⁡ ( π θ ) \mathrm{sg}(\pi_{\theta}) reweights which examples drive the update in a way explicit reward does not, yet leaves the loss basin unchanged.

#### Offline GRPO shifts direction but stays in basin; online GRPO does not.

Among offline rewards, only Offline GRPO substantially shifts direction from SFT (cosine 0.73 0.73 ; orthogonal-fraction 0.67 0.67 , ∼ 0.8 \sim\!\!0.8 late; angles up to 59 ∘ 59^{\circ} ), yet barrier-free interpolations keep it in the SFT/RIFT basin. Online GRPO goes much further — orthogonal fraction 0.998 0.998 (Figure 1 ) — so the SFT/offline-RL convergence is partly an artifact of shared fixed rollouts, which on-policy sampling breaks.

#### DPO sits apart, geometrically and on accuracy.

DPO occupies a near-orthogonal subspace ( 74 ∘ 74^{\circ} – 89 ∘ 89^{\circ} ), a higher-rank update with much smaller Frobenius norm, a sharp linear-mode barrier, and late-layer CKA ∼ 0.45 \sim 0.45 ; it also reaches the highest pass@1 on GSM8K ( 93.5 % 93.5\% ) and AIME26 ( 30.0 % 30.0\% ). It trains at a 10 × 10\times smaller LR, so its norm/accuracy gaps are entangled with the optimizer — correlation worth a LR-matched follow-up, not a causal claim. Extending the weight-space view to the wider contrastive family (IPO, KTO, SimPO, Cal-DPO) is left open.

#### Limitations.

Single domain and checkpoint; attention-only LoRA; greedy-only on small AIME26 ( n = 30 n{=}30 ). DPO uses 10 × 10\times smaller LR and ∼ 40 × {\sim}40\times fewer rows, so its norm/accuracy gaps are entangled with the optimizer. Online GRPO is reported at lr × 10 − 6 5\!\times\!10^{-6} , seed 42 42 (§ 3.4 ); the remaining LR/seed cells, a matched accuracy comparison, and calibrated DPO variants (KTO, IPO, SimPO) are left to a fuller sweep. Code, adapters, and analysis scripts are released.

## Appendix A Supplementary figures

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .