SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment — part 2/4

Cleaned-text code points [15518, 31514). Bibliography omitted here; source.md includes it.

We standardize rewards within each group to obtain scale-invariant learning signals: A k = ( r k − μ ) / ( σ + ϵ ) A_{k}=(r_{k}-\mu)/(\sigma+\epsilon) , where μ \mu and σ \sigma are the group mean and standard deviation. The training objective is: 𝒥 ( ϕ ) = 𝔼 x [ 1 K ∑ k = 1 K ρ k A k − β D KL ( π ϕ ∥ π ref ) ] \mathcal{J}(\phi)=\mathbb{E}_{x}\left[\frac{1}{K}\sum_{k=1}^{K}\rho_{k}A_{k}-\beta D_{\text{KL}}(\pi_{\phi}\|\pi_{\text{ref}})\right] (2) where ρ k = π ϕ ​ ( y k | x ) / π ϕ old ​ ( y k | x ) \rho_{k}=\pi_{\phi}(y_{k}|x)/\pi_{\phi_{\text{old}}}(y_{k}|x) is the importance ratio, ϕ old \phi_{\text{old}} denotes the parameters before the current update, and β \beta controls the KL penalty. Algorithm 1 summarizes the procedure.

##### Computational efficiency.

Computing stable rank requires O ⁡ ( T ​ d ) O(Td) operations: O ⁡ ( T ​ d ) O(Td) for the Frobenius norm and O ⁡ ( T ​ d ) O(Td) per power iteration for the spectral norm. This overhead is negligible compared to the transformer forward pass. Stable rank is also robust to input truncation: using only 512 tokens achieves nearly identical accuracy (Appendix H ).

## 4 Alignment Experiments

### 4.1 Experiment Setup

We evaluate SR-GRPO on two models: Qwen2.5-1.5B-Instruct ( Yang et al., 2024 ) and DeepSeek-R1-Distill-Qwen-1.5B ( DeepSeek-AI, 2025 ) . All methods are trained on prompts from SmolTalk2 ( Allal et al., 2025 ) , which provides diverse topics and tasks. No preference labels are used during training; methods differ only in how they compute rewards from sampled responses. For SR-GRPO, we train with LoRA ( Hu et al., 2022 ) and compute stable rank on the base model without LoRA adapters, which also serves as the frozen reference.

Benchmarks. We evaluate on three categories: (1) STEM tasks : GPQA ( Rein et al., 2024 ) , a graduate-level science QA benchmark requiring expert knowledge, and MMLU-redux ( Gema et al., 2025 ) , a curated subset of MMLU with corrected labels; (2) Mathematical reasoning : MATH500 ( Lightman et al., 2023 ) , a subset of competition mathematics problems, AIME25 ( Zhang and Math-AI, 2025 ) , problems from the 2025 American Invitational Mathematics Examination, OlympiadBench ( He et al., 2024b ) , olympiad-level math and physics problems, and AMC23 ( knoveleng, 2023 ) , problems from the 2023 American Mathematics Competition; (3) General chat : WildBench ( Lin et al., 2025 ) , which evaluates open-ended conversation quality using GPT-4o mini ( OpenAI, 2025 ) as a judge. For all benchmarks except WildBench, we report Pass@1 accuracy. WildBench ( Lin et al., 2025 ) evaluates open-ended conversation quality by comparing model outputs against reference responses using GPT-4o mini as a judge; scores are reported as Elo ratings.

Baselines. We compare SR-GRPO against two categories of methods:

(1) Reward Model (RM): We use Skywork-Reward-V2-Qwen3-1.7B ( Liu et al., 2025 ) , a widely used reward model trained on the Skywork preference dataset with the Bradley-Terry objective. We select a 1.7B model to match the scale of our policy models, ensuring fair comparison with self-evaluation methods that use the policy model itself for reward computation.

(2) Self-Evaluation Methods: These methods derive rewards from the model’s own outputs without external labels: • Self-Reward ( Yuan et al., 2024 ) : The model evaluates its own completions using the pointwise scoring prompt (Appendix B ), generating a 1-5 score that serves as the reward signal.

• Perplexity: We use the negative log-likelihood of the completion as the reward, where lower perplexity indicates higher fluency.

• IPO ( Garg et al., 2025 ) : The model acts as a binary classifier, determining response quality by generating “Yes” or “No”. We extract logits for both tokens from the first output position and compute their probabilities via softmax. The probability of “Yes” serves as the reward signal.

All methods use the same GRPO training configurations (K=8 responses per prompt, β \beta =0.01) to ensure fair comparison. Full training details are in Appendix E .

### 4.2 Results Analysis

Table 2 shows that SR-GRPO consistently outperforms both reward model and self-evaluation baselines across all categories, despite using no external labels.

Reasoning improvements. On Qwen2.5-1.5B-Instruct, SR-GRPO improves average mathematical reasoning accuracy by 4.4 percentage points (28.0% to 32.4%), with particularly strong gains on competition-level problems: AMC improves from 35.0% to 37.5% and MATH from 48.0% to 52.4%. On DeepSeek-R1-Distill-Qwen-1.5B, which already achieves 58.5% on math tasks, SR-GRPO still improves performance to 64.7%, demonstrating effectiveness even on reasoning-specialized models. The gains on OlympiadBench (43.6% to 58.5%) suggest that stable rank rewards the structured reasoning patterns required for complex problem-solving.

Comparison with baselines. The reward model shows minimal or even negative impact on several tasks (e.g., GPQA drops from 19.0% to 15.7% on Qwen2.5-1.5B), suggesting that models trained on general preference data may not transfer well to specialized reasoning domains. Self-evaluation methods show inconsistent patterns: Self-Reward matches SR-GRPO on AIME (13.3%) but underperforms on STEM tasks (31.6% vs 34.5% avg.); Perplexity performs surprisingly well on DeepSeek-R1 but lacks the consistency of SR-GRPO across both models. IPO degrades performance on several benchmarks, possibly because Yes/No probability signals are too coarse for nuanced quality distinctions.

General chat quality. SR-GRPO’s advantage extends beyond reasoning to open-ended conversation. On WildBench (WB-Elo), it achieves gains of 26.2 points on Qwen2.5-1.5B and 19.0 points on DeepSeek-R1, substantially outperforming all baselines. This indicates that stable rank captures qualities valued in general conversation, such as coherence and helpfulness, not just task-specific correctness. The consistent improvements across both reasoning benchmarks and open-ended chat suggest that stable rank provides a general-purpose quality signal rather than optimizing for narrow task metrics.

## 5 What Stable Rank Captures

To understand why stable rank can serve as an intrinsic reward, we analyze its correlations with interpretable text quality metrics on the RewardBench dataset. We compute 37 metrics spanning semantic coherence, information density, and linguistic structure. For each metric M M , we measure sample-level Pearson correlation with stable rank across 5,970 responses, and paired-difference Spearman correlation across 2,985 chosen/rejected pairs. The definitions and correlation tables are provided in Appendix F ; here we summarize key findings.

### 5.1 Stable Rank Captures Semantic Coherence

Table 3 presents the strongest correlations among semantic coherence metrics. Progression score ( ρ = 0.313 \rho=0.313 ) measures whether each sentence builds on the previous one. QA alignment consistency ( ρ = 0.316 \rho=0.316 ) captures whether relevance to the prompt remains stable throughout the response. Both correlate positively with stable rank. The strongest negative correlation appears with coherence standard deviation ( ρ = − 0.356 \rho=-0.356 ), where high values indicate erratic transitions between adjacent sentences. These patterns suggest stable rank can capture responses that maintain topical focus while developing ideas smoothly, avoiding the abrupt transitions characteristic of hallucinations or incoherent reasoning.

### 5.2 Stable Rank Distinguishes Density from Verbosity

Contrary to common reward hacking, where models maximize length, stable rank correlates negatively with token count ( ρ = − 0.294 \rho=-0.294 ) and sentence count ( ρ = − 0.368 \rho=-0.368 ). Instead, stable rank favors information density. Lexical diversity correlates positively ( ρ = 0.238 \rho=0.238 ). This metric measures the ratio of unique tokens to total tokens. Compression ratio also correlates positively ( ρ = 0.233 \rho=0.233 ). It measures the ratio of compressed to original text length. Table 3 presents key correlations. These patterns suggest stable rank penalizes verbose, repetitive text while rewarding concise responses with diverse vocabulary.

### 5.3 Stable Rank Highlights Key Logical Markers

In this section, we study the correlation between discourse markers and stable rank. Discourse markers are explicit phrases that signal logical relationships, such as “because” or “if”, or structural transitions, such as “however” or “first”. We use paired-difference analysis on N = 2,985 N=2{,}985 chosen and rejected pairs (Appendix F ) to correlate marker frequency, measured per 100 tokens, with stable rank while controlling for prompt context. We report both Pearson and Spearman correlations because the two can diverge when relationships are non-monotonic.

Table 4 shows that most marker categories, especially additive ( ρ = − 0.156 \rho=-0.156 ), conditional ( ρ = − 0.163 \rho=-0.163 ), and enumeration ( ρ = − 0.148 \rho=-0.148 ), have consistently negative correlations with stable rank, and the total marker count is also negatively associated ( ρ = − 0.204 \rho=-0.204 ). Responses that rely heavily on common connective words such as “moreover”, “first”, and “then” tend to receive a lower stable rank, which is consistent with Section 5.2 where stable rank disfavors verbose, pattern-like exposition in favor of compact information content.

Contrastive and causal markers follow a different pattern. Their Pearson correlations with stable rank are positive (contrastive r = 0.187 r=0.187 , causal r = 0.139 r=0.139 ), but their Spearman correlations are small. This divergence suggests that the presence of these markers matters more than their frequency: a few well-placed “however” or “because” phrases benefit stable rank, but overusing them does not. This aligns with linguistic intuition: contrastive and causal connectives are most informative when they mark key reasoning steps, not when they appear as formulaic filler.

### 5.4 Summary

Together, these correlation patterns align with established findings in text quality research. The strong association with semantic coherence metrics mirrors neural coherence models, which show that smooth semantic transitions predict text quality ( Mesgar and Strube, 2018 ; Cui et al., 2017 ) . The negative correlation with length while favoring information density addresses a well-documented failure mode in RLHF: length bias, where reward models favor longer responses regardless of quality ( Singhal et al., 2024 ) . Unlike learned reward models that often exploit this spurious correlation, stable rank’s geometric formulation inherently penalizes verbose, low-information content.

While these correlations are moderate in magnitude ( | ρ | ≈ 0.2 |\rho|\approx 0.2 – 0.4 0.4 ), they are consistent in direction across metrics and statistically significant ( p < 0.001 p<0.001 ), suggesting stable rank captures genuine quality signals rather than noise. These findings indicate that stable rank correlates with multiple dimensions of text quality, including coherence, density, and reasoning structure, suggesting it captures a meaningful aggregate signal rather than a single superficial feature. Qualitative examples illustrating how stable rank differentiates high-quality reasoning from repetition and verbosity are provided in Appendix J .

## 6 Ablation Studies

We conduct three ablation studies to validate our design choices.

Alternative intrinsic dimension metrics. We compare stable rank against three alternatives: condition number, which measures the ratio of largest to smallest singular value; PCA 95% variance, which counts principal components needed to capture 95% of variance; and effective rank ( Roy and Vetterli, 2007 ) , which uses entropy over the singular value distribution. On RewardBench, stable rank achieves 84.04% overall accuracy, outperforming PCA 95% variance (61.91%), effective rank (54.50%), and condition number (36.04%). The gap widens on harder categories such as Math and Safety. Stable rank succeeds because it aggregates information across the entire singular value spectrum through the Frobenius norm, making it robust to outliers. Condition number is sensitive to extreme singular values. Effective rank’s entropy weighting and PCA’s discrete component counting appear less suited for capturing quality distinctions. Full results are provided in Appendix G .

Context length. We examine how sequence truncation affects performance. Accuracy drops from 83.85% at 512 tokens to 62.59% at 128 tokens. Code suffers the largest degradation (87.91% to 24.80%) because truncation removes critical program logic. However, extending beyond 512 tokens yields negligible gains ( < < 0.2 percentage points). This indicates that stable rank captures core semantic structure rather than mechanically rewarding longer sequences. Details are provided in Appendix H .

Prompt format. We test six input templates across three models. Overall accuracy varies by at most 3 percentage points, with no format consistently outperforming others. This robustness simplifies deployment: practitioners can use simple formats without extensive tuning. Full results are shown in Appendix I .

## 7 Related Work

We review three lines of related work: alignment methods using external feedback, approaches that reduce reliance on human labels, and theoretical connections between representation geometry and generation quality.

### 7.1 Alignment with External Feedback Signals

The standard pipeline for aligning Large Language Models with human preferences is Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ; Bai et al., 2022 ) . RLHF trains a reward model on human preference data and then fine-tunes the policy to maximize this learned signal under a KL constraint to a reference model. This approach underpins LLMs such as ChatGPT and Llama ( Dubey et al., 2024 ) , but it inherits well-known issues: preference data are expensive and noisy, and reward models are vulnerable to reward hacking and distribution shift ( Casper et al., 2023 ; Skalse et al., 2022 ) . These limitations motivate supervised preference-based methods such as Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) , KTO ( Ethayarajh et al., 2024 ) , Bayesian RLHF ( Wang et al., 2023 ) , and MaxMin-RLHF ( Chakraborty et al., 2024 ) , which optimize the policy directly on pairwise comparisons without an explicit value network but still rely on curated preference datasets. Process reward models and generative reward models provide step-level scores and textual critiques for mathematical reasoning and general evaluation ( Zhang et al., 2025b ; Yin et al., 2025 ) .All these approaches keep supervision external to the model’s internal computation.

### 7.2 Alignment with Automatic Signals

A complementary line of work reduces reliance on human labels by using automatic supervision or signals derived from the model itself. Verifiable-reward methods such as DeepSeek-R1 ( DeepSeek-AI, 2025 ) and TÜLU 3 ( Lambert et al., 2024 ) replace human feedback with programmatic checkers; these methods work well in domains like mathematics and coding but do not generalize to open-ended tasks. Self-evaluation approaches treat the model, or another LLM, as a judge: Self-Rewarding Language Models ( Yuan et al., 2024 ) and RLAIF ( Lee et al., 2024 ) generate AI feedback instead of human labels, while IPO ( Garg et al., 2025 ) and Self-Rewarding PPO ( Zhang et al., 2025a ) derive preference signals from token-level probabilities. Closer to our setting, work on internal activations such as Factoscope ( He et al., 2024c ) and INSIDE ( Chen et al., 2024 ) shows that hidden states encode factuality and hallucination risk. Our method follows this direction, using a geometric statistic of activations as a reward signal for policy optimization.

### 7.3 Representation Geometry and Generation Quality