SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment — part 3/4

Cleaned-text code points [31514, 47385). Bibliography omitted here; source.md includes it.

A theoretical foundation for rank-based quality measures comes from the softmax bottleneck analysis. Yang et al. (2018) show that the expressiveness of softmax-based models is fundamentally limited by the rank of their hidden representations. Godey et al. (2024) extend this to modern LLMs, demonstrating that the contextual probability distribution of natural language is inherently high-rank. Complementary work shows that when embeddings collapse into a narrow cone during training, generation quality degrades ( Gao et al., 2019 ) . Together, these results establish a link between representational dimensionality and generation quality. Empirical work supports this connection. Garrido et al. (2023) demonstrates that the effective rank of learned representations predicts downstream performance without labels. In language models specifically, hidden states encode rich semantic structure: word embeddings capture syntactic and semantic regularities through geometric relationships ( Mikolov et al., 2013 ) , and intermediate representations encode factuality ( He et al., 2024c ) and hallucination risk ( Chen et al., 2024 ) , confirming that generation quality leaves detectable geometric signatures.

Our work builds on these findings but shifts from diagnosis to optimization. Rather than using rank to analyze pretrained models post-hoc, we employ stable rank as a live reward signal during policy optimization, transforming a geometric insight into a practical alignment method.

## 8 Conclusion

This work demonstrates that stable rank, a simple intrinsic geometric property of LLM hidden states, provides an effective signal for text quality assessment and model alignment. As a zero-shot reward proxy, stable rank achieves 84.04% accuracy on RewardBench and improves Best-of-N sampling by 11.3 percentage points on average across reasoning benchmarks. Building on these insights, we introduce SR-GRPO, which uses stable rank as a novel reward signal for reinforcement learning without external supervision. SR-GRPO improves Qwen2.5-1.5B-Instruct by 10% on STEM tasks and 19% on mathematical reasoning, outperforming both learned reward models and self-evaluation baselines. Moreover, our analysis reveals that stable rank captures semantic coherence, information density, and sensitivity to key reasoning structures. Collectively, these results suggest that internal representation geometry provides sufficient signal for effective LLM alignment, offering a scalable alternative to annotation-dependent approaches.

## Appendix A Cross-Layer Stable Rank Analysis

We analyze how stable rank performance varies across transformer layers by computing R SR ( l ) ​ ( y ) = ‖ 𝐇 l ‖ F 2 / ‖ 𝐇 l ‖ 2 2 R_{\text{SR}}^{(l)}(y)=\|\mathbf{H}_{l}\|_{F}^{2}/\|\mathbf{H}_{l}\|_{2}^{2} at each layer l l and evaluating it as a reward proxy on RewardBench ( Lambert et al., 2025 ) . We test three model families: Qwen3-8B (36 layers) ( Yang et al., 2025 ) , Qwen2.5-1.5B-Instruct (28 layers) ( Yang et al., 2024 ) , and Phi-3.5-mini-instruct (32 layers) ( Abdin et al., 2024 ) , measuring accuracy on five subcategories: Chat, Chat Hard, Safety, Code, and Math.

##### Final layers encode quality signals most effectively.

Figure 3 shows that stable rank performance varies dramatically across layers. Early and middle layers (indices 5-25) achieve near-random performance ( ∼ \sim 50% accuracy) across all categories, indicating that stable rank computed from these layers carries minimal quality information. In contrast, final layers (indices 28-32 depending on the model) exhibit sharp performance improvements, with accuracy rising to 70-85% on most categories. This pattern holds consistently across all three model families, suggesting that semantic abstraction in deeper layers is crucial for stable rank to capture response quality.

##### Task-dependent layer sensitivity.

While final layers universally outperform earlier layers, we observe task-specific patterns. On Chat and Chat Hard categories, performance plateaus across middle layers before rising sharply at the end, suggesting that conversational quality signals emerge primarily in final layers. For Safety, we observe an interesting U-shape on Qwen3-8B and Phi-3.5-mini, with both very early (layer 0-2) and final layers achieving reasonable performance. This may indicate that safety-related features are encoded at multiple levels of abstraction. Math and Code categories show the most dramatic improvements in final layers, with Qwen2.5-1.5B achieving near-perfect accuracy (95-98%) on Math using layer 27, highlighting that complex reasoning quality is best captured by the deepest semantic representations.

##### Implications for SR-GRPO.

These findings strongly support our design choice in SR-GRPO to extract stable rank from the final transformer layer. The consistent performance advantage of final-layer stable rank (70-85% accuracy) over middle layers (45-50% accuracy) demonstrates that quality-indicative geometric structure emerges primarily in deep semantic representations. This validates that maximizing final-layer stable rank during RL optimization directly targets the representational regime where quality signals are most pronounced, enabling effective annotation-free alignment.

## Appendix B Baseline Implementation Details

In this section, we provide the prompts and implementation details for all the baselines.

### B.1 Generative Baselines

In our experiments, we have two generative baselines: (1) Pointwise Scoring ( Kim et al., 2024 ) and (2) Pairwise Comparison ( Kim et al., 2024 ) .

#### B.1.1 Pointwise Scoring Prompt

For the Pointwise Scoring baseline, we use the following prompt structure to score each response on a 1-5 scale:

We extract the numerical rating from the model’s output by parsing the [RESULT] token and compare scores to determine the preferred response. The reference answer and score rubrics are adapted from RewardBench’s provided materials.

#### B.1.2 Pairwise Comparison Prompt

For the Pairwise Comparison baseline, we prompt the model to directly compare two responses:

We parse the model’s output to extract the preference (A or B) from the [RESULT] token. To account for positional bias, we evaluate each pair twice with responses swapped and only count predictions where both orderings agree.

### B.2 IPO Implementation

We follow the evaluation protocol from IPO ( Garg et al., 2025 ) for RewardBench. IPO computes implicit preference scores by prompting the model with task-specific Yes/No verification questions and extracting token probabilities from the final logits.

#### B.2.1 Task-Specific Evaluation Prompts

We use category-specific verification prompts adapted from the IPO codebase. All prompts follow the format: [verification_prompt] User: { instruction } Response: { response }. We automatically assign categories based on subset names: Code tasks contain ”hep-” or ”code”, Math tasks contain ”math-” or ”prm”, Safety tasks contain ”refusals-”, ”xstest-”, or ”donotanswer”, and all others default to Chat. The prompt structure varies by task:

Code: ”Below is a coding-related question along with a response containing code. Your task is to verify whether the provided code is correct, functional, and aligned with the requirements of the question. Answer with just Yes/No”

Math: ”Below is a math question along with a response containing a solution. Your task is to verify whether the solution is correct, the reasoning is logical, and the final answer is accurate. Answer with just Yes/No”

Safety: ”Inspect the response to ensure it is safe and appropriate for the question below. Answer Yes/No.”

Chat (default): ”Below is a question, along with a response to that question. Your task is to verify whether the response is helpful, factually correct, logically sound, and contextually appropriate. Answer with just Yes/No”

#### B.2.2 Score Computation

Given response y y to instruction x x , we tokenize the verification prompt and perform a forward pass to obtain logits ℓ ∈ ℝ | V | \boldsymbol{\ell}\in\mathbb{R}^{|V|} at the final position. We extract logits for ”Yes” and ”No” tokens (including variants with leading spaces), apply softmax normalization, and define the IPO score as: IPO ​ ( y | x ) = exp ⁡ ( ℓ Yes ) exp ⁡ ( ℓ Yes ) + exp ⁡ ( ℓ No ) \text{IPO}(y|x)=\frac{\exp(\ell_{\text{Yes}})}{\exp(\ell_{\text{Yes}})+\exp(\ell_{\text{No}})} (3)

The response with higher IPO ​ ( y | x ) \text{IPO}(y|x) is predicted as preferred. This approach differs from standard likelihood-based methods by explicitly prompting for binary quality judgments rather than computing sequence log-probabilities.

## Appendix C RewardBench Results

We show the complete performance comparison across different reward methods for RewardBench in Table 5 .

## Appendix D Best-of-N Complete Results

We provide complete Best-of-N decoding results comparing stable rank selection against random selection across four models and five benchmarks. For each configuration, we sample N candidate responses and either select randomly (Random@N) or choose the response with highest stable rank (Best@N). We report two relative metrics: Δ \Delta Rand. measures improvement over random selection at the same N, and Δ \Delta @1 measures improvement over greedy decoding (N=1).

Table 6 shows that stable rank selection consistently outperforms random selection across all models and sampling budgets. The advantage is most pronounced on smaller models: Llama-3.2-1B achieves +33.8% relative improvement over random selection at N=16, while random selection actually degrades performance below greedy decoding ( − - 9.9%). This degradation occurs because random selection may choose low-quality samples from the candidate pool, whereas stable rank reliably identifies better responses.

Notably, the gains from stable rank selection increase with N for most models, suggesting that larger candidate pools provide more opportunities for quality differentiation. On math-heavy benchmarks (MATH500, OlympiadBench), stable rank selection yields particularly strong improvements, with DeepSeek-R1 improving from 82.8% to 86.8% on MATH500 and from 43.6% to 59.7% on OlympiadBench at N=16.

## Appendix E SR-GRPO Training Details

We provide the training configurations for SR-GRPO experiments on both models. All experiments are conducted on 4 × \times NVIDIA H800 GPUs.

##### Qwen2.5-1.5B-Instruct.

We train for 400 steps using LoRA with rank r r =16 and α \alpha =32, targeting all attention and MLP projection layers. We use a learning rate of 10 − 6 10^{-6} with cosine scheduling and 10% warmup. Training uses per-device batch size of 4 with gradient accumulation steps of 8 (effective batch size 128), mixed precision (bfloat16), and gradient checkpointing for memory efficiency. The maximum prompt and completion lengths are both set to 4096 tokens.

##### DeepSeek-R1-Distill-Qwen-1.5B.

We train for 300 steps using the same LoRA configuration ( r r =16, α \alpha =32). Due to the model’s longer reasoning traces, we use per-device batch size of 2 with gradient accumulation steps of 16 (effective batch size 128). The maximum prompt length is set to 8192 and maximum completion length to 4096 tokens. All other hyperparameters remain identical to the Qwen configuration.

##### Stable Rank Computation.

For reward calculation, we format the prompt and completion using the model’s chat template. We then compute hidden states from the last transformer layer using the reference model. To prevent reward hacking, we temporarily disable LoRA adapters during stable rank computation, ensuring rewards are derived from the frozen base model’s representations rather than the adapting policy. Hidden states are extracted only for non-padding tokens, and stable rank is computed using the method described in Section 2 .

##### Common Settings.

Both models are trained on the SmolTalk2 ( Allal et al., 2025 ) preference dataset using paged AdamW optimizer with 8-bit quantization. We set weight decay to 0, maximum gradient norm to 1.0, and LoRA dropout to 0.1. Rewards are not normalized across batches. For GRPO-specific hyperparameters, we use the default values from the HuggingFace TRL library: group size K = 8 K=8 , KL penalty coefficient β = 0.04 \beta=0.04 , and numerical stability constant ϵ = 10 − 8 \epsilon=10^{-8} for advantage standardization.

## Appendix F Experimental Details for Metric Analysis

We analyze the relationship between stable rank and text quality using the RewardBench dataset ( Lambert et al., 2025 ) , which contains 2,985 prompts, each with one chosen and one rejected response ( N = 5,970 N=5{,}970 responses). This appendix describes our statistical methodology, defines all metrics, and reports the complete correlation results.

### F.1 Statistical Methodology

We employ two complementary analyses to characterize the relationship between stable rank and text metrics:

##### Sample-level correlation.

We treat each of the N = 5,970 N=5{,}970 responses as an independent observation. For metric M M and stable rank S S , we compute Pearson correlation r = corr ⁡ ( M , S ) r=\mathrm{corr}(M,S) and Spearman rank correlation ρ \rho . This measures whether higher stable rank globally associates with higher (or lower) metric values. We apply this method to semantic coherence, information density, and basic structural metrics.

##### Paired-difference analysis.

Linguistic markers (discourse and logical connectives) exhibit strong context dependence: some prompts naturally require more explanation markers than others, making absolute counts across different prompts statistically noisy. To control for this variation, we work at the pair level. For each prompt i i with chosen response y c y_{c} and rejected response y r y_{r} , we compute differences Δ ​ M i = M ⁡ ( y c ) − M ⁡ ( y r ) , Δ ​ S i = S ⁡ ( y c ) − S ⁡ ( y r ) , \Delta M_{i}=M(y_{c})-M(y_{r}),\quad\Delta S_{i}=S(y_{c})-S(y_{r}), (4)

and correlate Δ ​ M \Delta M with Δ ​ S \Delta S across all N = 2,985 N=2{,}985 pairs. This measures whether, within the same context, the response with higher stable rank tends to use more (positive correlation) or fewer (negative correlation) of a given linguistic feature.

### F.2 Metric Definitions

We measure text quality along three axes: semantic coherence, information density, and linguistic structure. Tables 8 – 10 define the metrics formally. For response y y , let T T denote its token sequence ( N tokens N_{\text{tokens}} tokens), s 1 , … , s n s_{1},\dots,s_{n} its sentences with embeddings v 1 , … , v n v_{1},\dots,v_{n} (from all-MiniLM-L6-v2 ( Reimers and Gurevych, 2019 ) ), and v p v_{p} the prompt embedding.

### F.3 Implementation Details

##### Sentence embeddings.

We use the all-MiniLM-L6-v2 ( Reimers and Gurevych, 2019 ) model from the sentence-transformers library, which produces 384-dimensional embeddings. The model applies mean pooling over token embeddings and L2-normalization.

##### Sentence tokenization.

We tokenize responses into sentences using NLTK 1 1 1 https://www.nltk.org/ , with a regex-based fallback (splitting on sentence boundaries while handling common abbreviations). Each response contains n n sentences indexed 1 , … , n 1,\dots,n .

##### Syllable counting.

For readability metrics, we count syllables using the pyphen 2 2 2 https://pyphen.org/ library (dictionary-based hyphenation for English). Complex words are defined as having ≥ 3 \geq 3 syllables following standard practice.

##### Marker detection.

Logical and discourse markers are detected via case-insensitive substring matching. For each marker category, we report: (1) raw count, (2) count normalized per sentence, and (3) count normalized per 100 tokens as ( c / N tokens ) × 100 (c/N_{\text{tokens}})\times 100 .

### F.4 Correlation Results