Weight-Space Geometry of Offline Reasoning Training — part 1/2

Cleaned-text code points [0, 15840). Bibliography omitted here; source.md includes it.

# Weight-Space Geometry of Offline Reasoning Training

###### Abstract

Offline reinforcement-learning losses (RFT, RIFT, DFT, Offline GRPO, DPO) are widely used to distill reasoning from large teachers into smaller students, and are typically compared on downstream accuracy alone. We ask whether they are mechanistically distinct or converge to a similar weight update. Training six methods (SFT, RFT, DFT, RIFT, Offline GRPO, DPO) on identical math rollouts from a single base model (Qwen3-4B) with attention-only LoRA, we analyze the resulting deltas via cosine similarity, principal-angle subspace analysis, linear mode connectivity, and CKA. We observe: (i) SFT, RFT, and RIFT have nearly colinear weight deltas (cosine ≥ 0.97 \geq 0.97 , top- 1 1 principal angle ∼ 7 ∘ {\sim}\!\!7^{\circ} median over 144 144 modules) and comparable GSM8K accuracy ( 87 87 – 88 % 88\% , n = 1319 n{=}1319 ; pairwise McNemar p ≥ 0.15 p\geq 0.15 ); (ii) DFT diverges further in direction than any reward-weighted method despite using the same data; (iii) Offline GRPO adds a substantial component orthogonal to the SFT direction ( ∼ 67 % \sim\!\!67\% globally, up to ∼ 86 % \sim\!\!86\% in late layers) while staying in the SFT loss basin; (iv) DPO sits in a near-orthogonal subspace, shows a mode-connectivity barrier, and collapses late-layer CKA to ∼ 0.46 \sim 0.46 . DPO also reaches the highest accuracy in our protocol on both GSM8K ( 93.5 % 93.5\% , McNemar p < 10 − 9 p<10^{-9} vs. each other method) and AIME26 ( 30.0 % 30.0\% vs. 3.3 3.3 – 10.0 % 10.0\% ); its training uses a 10 × 10\times smaller learning rate than the others (the standard convention), so the update-norm and accuracy gaps reflect loss-function and optimizer choices jointly, and a learning-rate-matched DPO comparison is left for future work.

###### Keywords:

## 1 Introduction

Reasoning distillation has become a standard recipe for teaching small models to solve math and code tasks: a strong teacher generates rollouts, and a student is trained on them with one of a rapidly growing list of offline objectives. The past year alone introduced RIFT ( Liu et al., 2026 ) , Offline GRPO ( KRAFTON AI, 2025 ) , DFT ( Wu and others, 2025 ) , LUFFY ( Yan et al., 2025 ) , and DAPO ( Yu and others, 2025 ) , alongside an established preference-learning family — DPO ( Rafailov et al., 2023 ) , KTO ( Ethayarajh et al., 2024 ) , IPO ( Azar et al., 2023 ) , and NCA ( Chen et al., 2024 ) — each accompanied by claims that its specific loss formulation is responsible for accuracy gains over plain SFT.

These methods are compared almost exclusively by benchmark accuracy. What they do to the model is unknown: do different losses produce weight updates that point in the same direction, or qualitatively different ones? The distinction matters for both practitioners (which loss is worth implementing?) and interpretability researchers (does “offline RL” name a single mechanism or a family?).

We present a controlled weight-space comparison of offline reasoning losses: identical rollouts, identical base model (Qwen3-4B-Instruct), shared LoRA initialization, six methods (DPO uses a smaller learning rate per its codebase convention, see § 2 ). Following recent weight-space studies of fine-tuning ( Arturi and others, 2025 ; Soligo and others, 2025 ; Zhong and Raghunathan, 2025 ; Ward and others, 2025 ) , we analyze each method’s LoRA delta Δ ​ W \Delta W rather than its outputs.

Our contributions are: (1) reward-weighted losses (SFT, RFT, RIFT) converge on essentially the same direction in weight space (cosine ≥ 0.97 \geq 0.97 ) and produce GSM8K accuracies that are non-different by exact McNemar’s test ( p ≥ 0.15 p\geq 0.15 , n = 1319 n{=}1319 ); (2) DFT, despite being a one-line modification of SFT, produces a more distinctive update than any explicitly reward-weighted method; (3) Offline GRPO adds a quantifiable orthogonal component (globally 67 % 67\% , rising to ∼ 80 % \sim\!\!80\% in late layers) while staying in the same loss basin as SFT/RIFT; (4) DPO sits in a near-orthogonal subspace with higher effective rank, a sharp linear-mode barrier, and reaches the highest pass@1 on both GSM8K and AIME26 in our protocol; we report this with the caveat that DPO uses a 10 × 10\times smaller learning rate, so the loss formulation and optimizer setting cannot be cleanly separated here.

## 2 Setup

#### Data.

All methods share one set of rollouts: DeepScaleR prompts ( ∼ 40 {\sim}40 k verified math ( Agentica, 2025 ) ), teacher DeepSeek-V4-Flash, K = 4 K{=}4 CoT completions/prompt, binary math-verify reward. Reference-policy methods use π base \pi_{\text{base}} . DPO consumes ∼ 1.8 {\sim}1.8 K (chosen, rejected) pairs vs. ∼ 75 {\sim}75 K rows for the rest. Identical rollouts is the central control.

#### Methods.

Table 1 summarizes the six losses. With ℓ i = − log ⁡ π θ ​ ( y i ∣ x ) \ell_{i}=-\log\pi_{\theta}(y_{i}\mid x) shorthand for the per-sequence NLL: SFT = ∑ i ℓ i =\sum_{i}\ell_{i} on all i i ; RFT ( Yuan et al., 2023 ) = ∑ i : r i = 1 ℓ i =\sum_{i:r_{i}=1}\ell_{i} on positives only; DFT ( Wu and others, 2025 ) = ∑ t sg ⁡ ( π θ ​ ( y t ∣ y < t , x ) ) ​ ℓ t tok =\sum_{t}\mathrm{sg}(\pi_{\theta}(y_{t}\mid y_{<t},x))\,\ell_{t}^{\text{tok}} down-weights confident tokens; RIFT ( Liu et al., 2026 ) = ∑ i ( 1 − r i ​ λ ) ​ ℓ i =\sum_{i}(1-r_{i}\,\lambda)\,\ell_{i} is a linear-in-reward surrogate that admits negatives; Offline GRPO ( KRAFTON AI, 2025 ; Shao et al., 2024 ) = ∑ i A ^ i ​ ℓ i =\sum_{i}\hat{A}_{i}\,\ell_{i} with A ^ i = r i − r ¯ g + b \hat{A}_{i}=r_{i}-\bar{r}_{g}+b ; DPO ( Rafailov et al., 2023 ) = − log ⁡ σ ⁡ ( β ⁡ [ log ⁡ π θ ​ ( y w ) π ref ​ ( y w ) − log ⁡ π θ ​ ( y l ) π ref ​ ( y l ) ] ) =-\log\sigma\!\left(\beta\,[\log\frac{\pi_{\theta}(y_{w})}{\pi_{\text{ref}}(y_{w})}-\log\frac{\pi_{\theta}(y_{l})}{\pi_{\text{ref}}(y_{l})}]\right) on (chosen y w y_{w} , rejected y l y_{l} ) pairs.

#### Training.

Qwen3-4B-Instruct-2507, LoRA on attention projections ( q,k,v,o_proj ; rank 32 32 , α = 64 \alpha{=}64 , dropout 0 0 ; 144 144 modules over 36 36 layers). Effective batch 32 32 , cosine schedule, 5 % 5\% warmup, wd 0.01 0.01 , grad-clip 1.0 1.0 , seed 42 42 , bf16. Peak LR × 10 − 6 5\!\times\!10^{-6} for all but DPO ( × 10 − 7 5\!\times\!10^{-7} , codebase convention; higher diverges). DPO: sigmoid loss, β = 0.1 \beta{=}0.1 ; Offline GRPO: additive bias 0.1 0.1 on the centered advantage, no probability weighting, no explicit KL. 1,500 1{,}500 steps; we report step 1,000 1{,}000 uniformly.

#### Analysis.

Let Δ ​ W ( m ) = B ( m ) ​ A ( m ) \Delta W^{(m)}=B^{(m)}A^{(m)} be the stacked LoRA delta of method m m . We measure: (i) global/per-layer cosine ⟨ Δ ​ W ( m ) , Δ ​ W ( m ′ ) ⟩ / ‖ Δ ​ W ( m ) ‖ ​ ‖ Δ ​ W ( m ′ ) ‖ \langle\Delta W^{(m)},\Delta W^{(m^{\prime})}\rangle/\|\Delta W^{(m)}\|\|\Delta W^{(m^{\prime})}\| ; (ii) per-layer SVD (effective rank, principal angles between top- k k subspaces); (iii) linear mode connectivity ( Frankle et al., 2020 ) : masked-answer CE on GSM8K along α ​ Δ ​ W ( m ) + ( 1 − α ) ​ Δ ​ W ( m ′ ) \alpha\Delta W^{(m)}+(1{-}\alpha)\Delta W^{(m^{\prime})} ; (iv) CKA ( Kornblith et al., 2019 ) of merged-model hidden states.

## 3 Results

### 3.1 Downstream accuracy

Figure 3 (appendix) reports greedy pass@1 on full GSM8K ( n = 1319 n{=}1319 ) and AIME26 ( n = 30 n{=}30 ). SFT/RFT/DFT/RIFT/Offline GRPO sit at 87.3 87.3 – 88.2 % 88.2\% on GSM8K, pairwise non-different by exact McNemar ( p ≥ 0.15 p\geq 0.15 ); DPO reaches 93.5 % 93.5\% ( p < 10 − 9 p<10^{-9} ). On AIME26 the ordering repeats but n = 30 n{=}30 is underpowered (SFT–DPO p = 0.07 p{=}0.07 ). DPO trains at a 10 × 10\times smaller LR with ∼ 40 × {\sim}40\times fewer rows, so we treat the gap as suggestive. Llama-3.2-3B replicates the geometry and the 5 5 – 7 7 point DPO accuracy edge.

#### On-policy RL preserves accuracy; SFT-style loses it.

Re-measuring greedy pass@1 for the on-policy methods (Table 2 , our adapters, same protocol) shows their reward-orthogonal updates (§ 3.4 ) do not cost accuracy: Online GRPO/DAPO and DPO all stay at the base instruct model’s 93 93 – 94 % 94\% on GSM8K, whereas SFT and Offline GRPO drop to ∼ 87 % {\sim}87\% (below base). Online GRPO is best on AIME26 ( 20.0 % 20.0\% ).

### 3.2 Weight-space convergence

Figure 1 shows the cosine similarity matrix between global LoRA deltas. Three regimes are visible. First , SFT, RFT, and RIFT form a tight cluster: the SFT–RFT, SFT–RIFT, and RFT–RIFT cosines are 0.977 0.977 , 0.967 0.967 , 0.969 0.969 . Filtering negatives (RFT) and reward-weighting them (RIFT) does not measurably change the direction of the update relative to plain SFT on the union; it only adjusts the step size, with ‖ Δ ​ W ‖ F \|\Delta W\|_{F} ranging from 2.82 2.82 (RFT) to 3.04 3.04 (RIFT). Second , DFT, which differs from SFT by a single multiplicative factor on the loss, sits at cosine 0.572 0.572 to SFT and 0.536 0.536 to RIFT — a larger directional change than any explicitly reward-weighted method. Third , DPO is orthogonal to everything: cosine to SFT, RFT, and RIFT all fall in [ 0.057 , 0.065 ] [0.057,0.065] . Offline GRPO occupies an intermediate position (cosine ≈ 0.74 \approx 0.74 to the SFT cluster).

The per-layer view (Figure 4 ) decomposes this further. SFT/RFT/RIFT pairs are essentially flat above 0.95 0.95 at every layer. SFT–GRPO drops gradually with depth. SFT–DFT is bimodal — close to 1 1 at certain bottleneck layers and well below 0.5 0.5 at others. SFT–DPO hovers near zero throughout. The visible drop on layers 34 34 – 36 36 across all pairs reflects the small-norm tail of LoRA updates in the last decoder block; we treat this as a LoRA artifact rather than a finding.

### 3.3 Subspace analysis

Per-layer SVD lets us go beyond a single direction and ask whether two methods adapt the same low-dimensional subspace. We report principal angles between the top- 10 10 left singular vectors of each Δ ​ W ( m ) \Delta W^{(m)} at the same module. Smaller angles mean shared subspace.

Aggregating across all 144 144 attention modules (top- 10 10 left singular vectors per module), median top- 1 1 principal angles are 6.7 ∘ 6.7^{\circ} (SFT–RFT), 8.2 ∘ 8.2^{\circ} (SFT–RIFT), 18.5 ∘ 18.5^{\circ} (SFT–Offline GRPO), 26.7 ∘ 26.7^{\circ} (SFT–DFT), and 54.6 ∘ 54.6^{\circ} (SFT–DPO); the median worst (top- 10 10 ) angles are 36 ∘ 36^{\circ} , 40 ∘ 40^{\circ} , 76 ∘ 76^{\circ} , 85 ∘ 85^{\circ} , 90 ∘ 90^{\circ} in the same order. SFT–DPO IQR for the worst angle is [ 89.6 ∘ , 89.8 ∘ ] [89.6^{\circ},89.8^{\circ}] : essentially every module is orthogonal at every singular index. The reward-weighted cluster shares the top of its subspace with SFT to within ∼ 10 ∘ {\sim}\!\!10^{\circ} ; GRPO and DFT partially overlap; DPO does not.

The effective rank, averaged over all 144 144 modules, is ∼ 16 \sim 16 for SFT, RFT, DFT, and RIFT, 14.8 14.8 for Offline GRPO, and 24.5 24.5 for DPO. DPO writes into a higher-dimensional subspace, but, given its 13 × 13\times smaller Frobenius norm, with much smaller singular values; together with the orthogonality to SFT, this suggests DPO learns a different decomposition of the same projection matrices rather than a low-rank refinement of SFT.

To quantify how much of Offline GRPO’s update is genuinely new direction, we project Δ ​ W grpo \Delta W^{\mathrm{grpo}} onto the SFT direction at every adapted module and report ‖ Δ ​ W grpo − Π sft ​ Δ ​ W grpo ‖ F / ‖ Δ ​ W grpo ‖ F \|\Delta W^{\mathrm{grpo}}-\Pi_{\mathrm{sft}}\Delta W^{\mathrm{grpo}}\|_{F}/\|\Delta W^{\mathrm{grpo}}\|_{F} . Globally this is 0.67 0.67 ; per layer it grows from ∼ 0.55 \sim\!\!0.55 in middle blocks to 0.79 0.79 – 0.86 0.86 in the final five blocks — the same layers where CKA diverges (Section 3.6 ).

#### Top-1 singular directions.

The rank-1 approximation Δ ​ W ≈ σ 1 ​ u 1 ​ v 1 ⊤ \Delta W\approx\sigma_{1}u_{1}v_{1}^{\top} isolates the single most important output direction u 1 u_{1} each loss writes into. Mean | ⟨ u 1 m , u 1 m ′ ⟩ | |\langle u_{1}^{m},u_{1}^{m^{\prime}}\rangle| over 144 144 modules is 0.97 0.97 – 0.98 0.98 within SFT/RFT/RIFT, 0.78 0.78 – 0.80 0.80 to Offline GRPO, 0.64 0.64 – 0.67 0.67 to DFT, and 0.11 0.11 to DPO. Right singular vectors v 1 v_{1} (input directions) converge much more tightly: 0.99 0.99 – 1.00 1.00 for non-DPO pairs, 0.94 0.94 for DFT, 0.66 0.66 – 0.70 0.70 for DPO. All methods (except DPO) read from nearly the same input subspace; they differ in how they transform it.

### 3.4 Seed and learning-rate sensitivity

The colinearity above is at a single seed, conflating loss agreement with shared-init agreement. We disentangle by training each loss at two seeds ( 42,123 42,123 ) and three LRs ( × 10 − 7 . . − 5 5\!\times\!10^{-7..-5} ); Δ ​ W = ( α / r ) ​ B ​ A \Delta W=(\alpha/r)BA is gauge-invariant, so its cosine is genuine.

Seed rotates Δ ​ W \Delta W more than the loss — but only on the input side. At a fixed seed SFT–RFT are colinear (cosine 0.996 0.996 , angle 3.7 ∘ 3.7^{\circ} ), yet the same loss at two seeds has cosine only 0.07 0.07 ( × 10 − 7 5\!\times\!10^{-7} )– 0.36 0.36 ( × 10 − 5 5\!\times\!10^{-5} ). Cause: LoRA’s random A A init — across seeds the top- 1 1 output direction u 1 u_{1} still agrees at 0.99 0.99 while the input direction v 1 v_{1} agrees at 0.07 0.07 (median top- 8 8 angle 26 ∘ 26^{\circ} vs. 76 ∘ 76^{\circ} for unrelated runs). Functionally the seeds are the same solution: interpolating their deltas shows no barrier (midpoint + 0.004 +0.004 ). So the cross-method colinearity is partly shared-init, but convergence onto a common output subspace is seed-robust (Figure 2 ).

Learning rate changes direction, not just magnitude. A 10 × 10\times LR step rotates Δ ​ W \Delta W (cosine ≈ 0.55 \approx 0.55 ) and grows its norm only ∼ 3 × \sim\!3\times — not a pure rescaling, which sharpens the caveat on the 10 × 10\times -smaller-LR DPO comparison.

Online GRPO is far more orthogonal than offline GRPO. We also train online GRPO under the same LoRA recipe (on-policy rollouts, group-relative advantage, math_verify reward; 600 600 steps, 8 8 generations/prompt, lr × 10 − 6 5\!\times\!10^{-6} , seed 42 42 ) — the comparison the original protocol could not produce. The resulting update is almost entirely orthogonal to the SFT/RFT cluster: cosine 0.025 0.025 to SFT and 0.024 0.024 to RFT, with an orthogonal fraction of 0.998 0.998 off the SFT direction (Figure 1 ), versus 0.67 0.67 for offline GRPO (§ 3.2 ). Its Frobenius norm is ∼ 10 × \sim\!10\times smaller than SFT’s at the same LR ( 0.30 0.30 vs. 2.84 2.84 ), echoing the small-norm regime of DPO. On-policy sampling thus moves the update off the shared SFT subspace far more than the offline group-relative loss does, indicating that the SFT/offline-RL directional convergence is partly a consequence of training on the same fixed rollouts : replacing them with on-policy samples largely breaks it.

### 3.5 Linear mode connectivity

We linearly interpolate LoRA deltas, merge into the base, and measure the per-token cross-entropy of the gold \ 𝚋𝚘𝚡𝚎𝚍 ​ { 𝚊𝚗𝚜𝚠𝚎𝚛 } \backslash\mathtt{boxed\{answer\}} continuation right after the prompt on GSM8K. The metric is length-sensitive: DPO produces longer, structured CoTs (median 5100 5100 vs 1100 1100 chars on correct AIME26; verification steps in 9 / 9 9/9 correct DPO solutions vs 0 / 3 0/3 SFT), inflating per-token NLL of the bare boxed answer. Offline GRPO → \to RIFT improves monotonically ( 4.93 → 2.25 4.93\to 2.25 ) and SFT → \to Offline GRPO worsens monotonically ( 2.06 → 4.93 2.06\to 4.93 ): one basin. RIFT → \to DPO shows a sharp non-monotonic barrier above α = 0.5 \alpha{=}0.5 ( 3.82 → 7.06 → 8.64 3.82\to 7.06\to 8.64 ): even discounting length, linear interpolation destroys the solution.

### 3.6 Representational similarity