RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics

arxiv:2604.01375

Original source · Paper JSON

Post-training · 2026-04-01

Cites (0) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics
• Abstract
• 1 Introduction
• 2 Related Work
• 3 The RubrIc Failure mode Taxonomy
• 4 Automated RIFT Evaluators
• 5 Conclusions
• 6 Limitations
• 7 Ethical Statement
• Appendix A Data Source and Sampling Provenance
• A.1 Data source distribution.
• A.2 Sampling.
• Appendix B Additional Experiment Details
• B.1 Alignment and Reward Variance Judge Model Ablation
• B.2 Additional Analysis for non-LLMaJ RIFT Evaluators
• Appendix C Prompts
• C.1 LLM-as-a-Judge Annotation Prompt
• C.2 Taxonomy Refinement Prompt
• Taxonomy Philosophy CRITICAL: This taxonomy will be used by human annotators. The primary goal is to create a taxonomy that is: • Compact : Aim for 7–10 total failure modes. Fewer distinct categories is ALWAYS better than many granular ones.
• Guidelines
• Task
• Appendix D RIFT: RubrIc Failure Mode Taxonomy
• D.1 Reliability Failures
• Subjective.
• Non-Atomic.
• Ungrounded.
• D.2 Content Validity Failures
• Misaligned or Rigid.
• Missing Criteria.
• D.3 Consequential Validity Failures
• Hackable.
• Low Signal.
• Redundant Criteria.
• Appendix E Qualitative Analysis of Failure of LLMaJ in Detecting Failure Modes
• Hackable : rewarding presentation instead of task success.
• Ungrounded : clear requirement, missing verification procedure.
• Instructions for reporting errors