SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment — part 1/4

Cleaned-text code points [0, 15518). Bibliography omitted here; source.md includes it.

# SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment

###### Abstract

Aligning Large Language Models (LLMs) with human preferences typically relies on external supervision, which faces critical limitations: human annotations are scarce and subjective, reward models are vulnerable to reward hacking, and self-evaluation methods suffer from prompt sensitivity and biases. In this work, we propose stable rank , an intrinsic, annotation-free quality signal derived from model representations. Stable rank measures the effective dimensionality of hidden states by computing the ratio of total variance to dominant-direction variance, capturing quality through how information distributes across representation dimensions. Empirically, stable rank achieves 84.04% accuracy on RewardBench and improves task accuracy by an average of 11.3 percentage points over greedy decoding via Best-of-N sampling. Leveraging this insight, we introduce Stable Rank Group Relative Policy Optimization (SR-GRPO) , which uses stable rank as a reward signal for reinforcement learning. Without external supervision, SR-GRPO improves Qwen2.5-1.5B-Instruct by 10% on STEM and 19% on mathematical reasoning, outperforming both learned reward models and self-evaluation baselines. Our findings demonstrate that quality signals can be extracted from internal model geometry, offering a path toward scalable alignment without external supervision.

## 1 Introduction

The alignment of Large Language Models (LLMs) with human preferences typically relies on Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ; Bai et al., 2022 ) . Despite its success, this paradigm depends heavily on external supervision. Human judgments are subjective and context-dependent, making it difficult to train robust reward models ( Chakraborty et al., 2024 ; Wang et al., 2023 ) , and learned proxies are susceptible to reward hacking ( Casper et al., 2023 ) . The sparsity of annotations further limits the model’s ability to fine-grained behaviors ( Wu et al., 2023 ) .

These limitations have motivated various attempts to reduce annotation dependence. Direct Preference Optimization ( Rafailov et al., 2023 ) eliminates explicit reward modeling but still requires preference datasets. Verifiable rewards ( DeepSeek-AI, 2025 ; Lambert et al., 2024 ) leverage ground-truth outcomes but only apply where automatic verification is feasible. Self-evaluation methods ( Yuan et al., 2024 ; Lee et al., 2024 ; Garg et al., 2025 ) suffer from prompt sensitivity and systematic biases ( Zheng et al., 2023 ; Wang et al., 2024 ) . Crucially, all these approaches still evaluate quality through external signals while overlooking information embedded in the model’s own representations.

Yet the validity of a generation should be reflected in its underlying computation. When a model outputs “The capital of France is Paris,” its hidden states encode activated factual knowledge ( He et al., 2024c ) ; when it hallucinates, different activation patterns emerge ( Chen et al., 2024 ) . This raises a natural question: Can we measure text quality directly from internal signals, enabling LLM alignment without external supervision?

In this work, we show that a simple geometric property of LLM hidden states, their effective dimensionality, provides a reliable signal for LLM response quality. We propose to use stable rank ( Rudelson and Vershynin, 2007 ) , a matrix-theoretic measure as an unsupervised proxy for generation quality. Stable rank quantifies how many independent semantic directions a response occupies by measuring the ratio of total variance to dominant-direction variance in the hidden-state representation. It balances representational richness with coherence: a high stable rank indicates that information spreads across many dimensions rather than concentrating in a few. The theoretical motivation comes from two lines of work. First, the softmax bottleneck analysis shows that accurate language modeling requires navigating a high-dimensional semantic manifold, as the contextual probability distribution of natural language is inherently high-rank ( Yang et al., 2018 ; Godey et al., 2024 ) . Second, literature also shows that when representations collapse into a narrow cone, expressiveness is severely limited and generation quality degrades ( Gao et al., 2019 ) . We therefore hypothesize that this principle extends to individual LLM responses: high-quality generations should maintain higher effective dimensionality, while low-quality outputs exhibit rank collapse.

We validate this hypothesis in two settings. First, as a zero-shot reward proxy, stable rank achieves 84.04% accuracy on RewardBench ( Lambert et al., 2025 ) with Qwen3-8B ( Yang et al., 2025 ) , matching LLM-as-Judge baselines without any training. Figure 1 illustrates this geometrically: while PCA projections of LLM representations show overlap between good and bad responses, stable rank cleanly separates them. Second, stable rank-guided Best-of-N selection consistently outperforms greedy decoding across four model families on STEM and mathematics benchmarks, with average gains of 11.3% at N = 16 N=16 .

Building on these findings, we introduce Stable Rank Group Relative Policy Optimization (SR-GRPO) , which uses stable rank as an intrinsic reward signal for reinforcement learning. By replacing external supervision with this geometric signal derived directly from the reference model, we enable fully annotation-free alignment. On Qwen2.5-1.5B-Instruct ( Yang et al., 2024 ) , SR-GRPO outperforms GRPO with learned reward models by 10 to 19% on reasoning tasks at zero annotation cost, and surpasses self-reward baselines by 8 to 9%. Consistent gains on DeepSeek-R1-Distill-Qwen-1.5B ( DeepSeek-AI, 2025 ) confirm robustness across model families.

To understand why stable rank succeeds as a quality signal, we analyze its correlations with interpretable text metrics. We find that stable rank captures three quality dimensions: semantic coherence, where sentences build on each other while staying relevant to the prompt; information density over verbosity, favoring concise, non-repetitive text; and sensitivity to reasoning words like “however” and “because” at key turning points.

Our contributions are summarized as follows: • We propose SR-GRPO, which uses stable rank as a dense reward signal in reinforcement learning, eliminating dependency on preference datasets or external verifiers.

• We demonstrate that SR-GRPO improves reasoning across multiple benchmarks, thus establishing intrinsic geometric signals as a viable basis for LLM alignment.

• Beyond reinforcement learning, we also show that stable rank offers a reliable zero-shot reward proxy for evaluating LLM responses, and integrates naturally into best-of-N decoding.

## 2 Stable Rank as an Intrinsic Quality Metric

We propose stable rank ( Rudelson and Vershynin, 2007 ) as an unsupervised quality signal derived from LLM representations. This section defines the metric, explains its geometric motivation, and validates its effectiveness as a reward proxy.

### 2.1 Definition

For a sequence of T T tokens, we extract the hidden state activation matrix 𝐇 ∈ ℝ T × d \mathbf{H}\in\mathbb{R}^{T\times d} from the last layer of an LLM. Let σ 1 ≥ σ 2 ≥ ⋯ ≥ σ min ⁡ ( T , d ) ≥ 0 \sigma_{1}\geq\sigma_{2}\geq\dots\geq\sigma_{\min(T,d)}\geq 0 denote the singular values of 𝐇 \mathbf{H} . The stable rank is defined as: SR ​ ( 𝐇 ) = ‖ 𝐇 ‖ F 2 ‖ 𝐇 ‖ 2 2 = ∑ i σ i 2 σ 1 2 . \text{SR}(\mathbf{H})=\frac{\|\mathbf{H}\|_{F}^{2}}{\|\mathbf{H}\|_{2}^{2}}=\frac{\sum_{i}\sigma_{i}^{2}}{\sigma_{1}^{2}}. (1) Stable rank measures the effective dimensionality of 𝐇 \mathbf{H} : if a single singular value dominates, SR ​ ( 𝐇 ) ≈ 1 \text{SR}(\mathbf{H})\approx 1 , indicating representation collapse; when singular values are balanced, stable rank approaches rank ​ ( 𝐇 ) \text{rank}(\mathbf{H}) , reflecting a rich, high-dimensional representation. Intuitively, low stable rank means token representations cluster along a few dominant directions, while a high stable rank means they spread across the embedding space.

##### Theoretical motivation.

The theoretical motivation comes from two lines of work. The softmax bottleneck ( Yang et al., 2018 ; Godey et al., 2024 ) implies that expressive language modeling requires high-rank representations, while representation collapse is a known symptom of degraded generation ( Gao et al., 2019 ) . Stable rank directly measures this effective dimensionality, making it a natural candidate for quality assessment. We validate this connection empirically below and through correlation analysis in Section 5 .

##### Implementation.

Given a prompt-response pair formatted with the model’s chat template, we perform a forward pass and extract the hidden state matrix 𝐇 ∈ ℝ T × d \mathbf{H}\in\mathbb{R}^{T\times d} from the final hidden layer, where T T is the total number of tokens. We use final-layer activations because they aggregate information across all preceding layers and are directly related to the final text generation. Stable rank is then computed from 𝐇 \mathbf{H} via Equation 1 . Cross-layer analysis in Appendix A confirms that the final layer embeddings yield the strongest quality signal.

### 2.2 Empirical Validation: Stable Rank as Zero-Shot Reward Proxy

We evaluate stable rank as a zero-shot proxy for human preference by testing whether it can predict the preferred response in a pair. This assessment checks whether stable rank can serve as a reliable reward signal without any model-specific training.

##### Setup.

We use RewardBench ( Lambert et al., 2025 ) , a benchmark containing 2,985 preference pairs across five categories: Chat, Chat-Hard, Safety, Code, and Math. For each pair, we compute stable rank for both responses and predict the one with the higher stable rank as preferred. We compare against three baseline methods: (1) Pointwise Scoring (Point.) ( Kim et al., 2024 ) : prompting the LLM to score each response on a 1-5 scale; (2) Pairwise Comparison (Pair.) ( Kim et al., 2024 ) : prompting the LLM to directly compare two responses; (3) IPO ( Garg et al., 2025 ) : using implicit preference scores derived from Yes/No token probabilities. We evaluate five models: Qwen2.5-1.5B-Instruct ( Yang et al., 2024 ) , Qwen3-0.6B ( Yang et al., 2025 ) , Qwen3-8B ( Yang et al., 2025 ) , Llama-3.1-8B-Instruct ( Dubey et al., 2024 ) , and Phi-3.5-mini-Instruct ( Abdin et al., 2024 ) . For stable rank, we format inputs as prompt-response pairs using each model’s native chat template, extract final-layer hidden states for the response tokens only, and compute stable rank via Eq. 1 . Prompts and implementation details are in Appendix B .

##### Results.

Table 1 shows stable rank consistently outperforms baselines across model scales. On Qwen3-8B, it achieves 84.04% accuracy, surpassing Pointwise (83.70%) and IPO (78.02%). The advantage is more pronounced on smaller models: stable rank achieves 75.95% on Qwen2.5-1.5B-Instruct, outperforming the best baseline (IPO, 65.85%) by 10.1 percentage points. Generative baselines exhibit high variance across scales. Pointwise scoring excels on large models (83.70% on Qwen3-8B) but collapses on smaller ones (37.15% on Qwen2.5-1.5B), likely because smaller models lack the instruction-following capability to produce calibrated scores. Pairwise comparison shows the opposite pattern, performing better on small models but degrading on Qwen3-8B (71.98%). IPO maintains relative stability (65.57%–78.02%) but still underperforms stable rank across all models. The consistent advantage of stable rank, especially on smaller models, suggests that LLM’s intrinsic geometric signals are more robust than prompt-based evaluation. Prompt-based methods require LLMs to follow evaluation instructions and produce calibrated scores, which smaller models often struggle with. Stable rank has no such dependency, making it particularly suitable for aligning tiny LLMs.

### 2.3 Empirical Validation: Best-of-N Decoding with Stable Rank

We next evaluate stable rank as a test-time quality signal in Best-of-N decoding, where the model samples multiple candidate responses and selects the one with the highest stable rank. This setup tests whether stable rank can guide inference and improve task performance beyond standard greedy decoding.

##### Setup.

For each prompt, we sample N ∈ { 1 , 4 , 8 , 16 } N\in\{1,4,8,16\} responses (temperature 0.7, top-p 0.9) and select the one with the highest stable rank. We test on STEM benchmarks (GPQA ( Rein et al., 2024 ) , MMLU-redux ( Gema et al., 2025 ) ) and mathematical reasoning tasks (MATH500 ( Lightman et al., 2023 ) , OlympiadBench ( He et al., 2024a ) , AMC23 ( knoveleng, 2023 ) ) across four models: Qwen2.5-1.5B-Instruct ( Yang et al., 2024 ) , Phi-3.5-mini-Instruct ( Abdin et al., 2024 ) , Llama-3.2-1B-Instruct ( Dubey et al., 2024 ) , and DeepSeek-R1-Distill-Qwen-1.5B ( DeepSeek-AI, 2025 ) . We compare against random selection to isolate the effect of the selection criterion from sampling diversity.

##### Results.

Figure 2 shows that stable rank-guided selection consistently improves over greedy decoding across all models. At N=16, Llama-3.2-1B achieves the largest gain (+20.5%), followed by Qwen2.5-1.5B (+17.0%) and DeepSeek-R1 (+10.2%), and Phi-3.5-mini shows a moderate improvement of +8.5%. Stable rank also outperforms random selection. On Llama-3.2-1B at N=16, stable rank (26.5% avg.) exceeds random selection (19.8%) by 33.8%. Notably, random selection often degrades performance below greedy decoding, such as − - 16.1% on Llama-3.2-1B at N=8. In contrast, stable rank consistently yields positive gains, demonstrating that it identifies high-quality responses rather than merely benefiting from sampling diversity. These results confirm that stable rank captures quality signals that correlate strongly with correctness. Complete results are provided in Appendix D .

## 3 Stable Rank Group Relative Policy Optimization (SR-GRPO)

Having established an empirical correlation between stable rank and response quality, we next ask whether using stable rank as a reward signal in reinforcement learning can further improve LLM alignment behavior. Because stable rank requires no labeled data and can be computed for any generated response, it provides a simple and scalable dense reward for reinforcement learning.

### 3.1 Setup

Let π ref \pi_{\text{ref}} denote a reference model and π ϕ \pi_{\phi} the trainable policy initialized from π ref \pi_{\text{ref}} . Given a prompt x x , the policy generates a response y ∼ π ϕ ( ⋅ | x ) y\sim\pi_{\phi}(\cdot|x) . Our goal is to optimize π ϕ \pi_{\phi} to generate higher-quality responses, using stable rank as a proxy reward.

### 3.2 Training Procedure

We build on Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) , which compares responses within a sampled group to obtain relative quality signals, avoiding the need for a learned value function. This avoids the need to train a separate critic and suits our setting where rewards come from a geometric metric.

For each prompt x x , we sample K K responses from the current policy and compute their stable rank on the frozen reference model π ref \pi_{\text{ref}} . Using a frozen model is critical: it provides a stationary reward signal that the policy cannot manipulate by changing its own internal geometry.