SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment

arxiv:2512.02807

Original source · Paper JSON

Post-training · 2025-12-02

Cites (3) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment
• Abstract
• 1 Introduction
• 2 Stable Rank as an Intrinsic Quality Metric
• 2.1 Definition
• Theoretical motivation.
• Implementation.
• 2.2 Empirical Validation: Stable Rank as Zero-Shot Reward Proxy
• Setup.
• Results.
• 2.3 Empirical Validation: Best-of-N Decoding with Stable Rank
• Setup.
• Results.
• 3 Stable Rank Group Relative Policy Optimization (SR-GRPO)
• 3.1 Setup
• 3.2 Training Procedure
• Computational efficiency.
• 4 Alignment Experiments
• 4.1 Experiment Setup
• 4.2 Results Analysis
• 5 What Stable Rank Captures
• 5.1 Stable Rank Captures Semantic Coherence
• 5.2 Stable Rank Distinguishes Density from Verbosity
• 5.3 Stable Rank Highlights Key Logical Markers
• 5.4 Summary
• 6 Ablation Studies
• 7 Related Work
• 7.1 Alignment with External Feedback Signals
• 7.2 Alignment with Automatic Signals
• 7.3 Representation Geometry and Generation Quality
• 8 Conclusion
• Appendix A Cross-Layer Stable Rank Analysis
• Final layers encode quality signals most effectively.
• Task-dependent layer sensitivity.
• Implications for SR-GRPO.
• Appendix B Baseline Implementation Details
• B.1 Generative Baselines
• B.1.1 Pointwise Scoring Prompt
• B.1.2 Pairwise Comparison Prompt
• B.2 IPO Implementation
• B.2.1 Task-Specific Evaluation Prompts
• B.2.2 Score Computation
• Appendix C RewardBench Results
• Appendix D Best-of-N Complete Results
• Appendix E SR-GRPO Training Details
• Qwen2.5-1.5B-Instruct.
• DeepSeek-R1-Distill-Qwen-1.5B.
• Stable Rank Computation.
• Common Settings.
• Appendix F Experimental Details for Metric Analysis
• F.1 Statistical Methodology
• Sample-level correlation.
• Paired-difference analysis.
• F.2 Metric Definitions
• F.3 Implementation Details
• Sentence embeddings.
• Sentence tokenization.
• Syllable counting.
• Marker detection.
• F.4 Correlation Results
• F.5 Key Findings
• Appendix G Ablation: Comparison with Other Intrinsic Dimension Metrics
• Metrics.
• Results.
• Analysis.
• Appendix H Ablation: Context Window Size
• Results.
• Analysis.
• Practical Recommendations.
• Appendix I Ablation: Input Prompt Format
• Results.
• Practical Recommendations.
• Appendix J Stable Rank Example Prompts
• Catastrophic Repetition.
• Redundant Verbosity.
• Instructions for reporting errors