RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics — part 1/4

Cleaned-text code points [0, 15884). Bibliography omitted here; source.md includes it.

# RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics

###### Abstract

Rubric-based evaluation is widely used in LLM benchmarks and training pipelines for open-ended, less verifiable tasks. While prior work has demonstrated the effectiveness of rubrics using downstream signals such as reinforcement learning outcomes, there remains no principled way to diagnose how a rubric itself fails from such aggregated or downstream signals alone. To address this gap, we introduce RIFT: RubrIc Failure mode Taxonomy , a taxonomy for systematically characterizing failure modes in rubric composition and design. RIFT consists of eight failure modes organized into three high-level categories: Reliability Failures , Content Validity Failures , and Consequential Validity Failures . RIFT is developed using grounded theory by iteratively annotating rubrics drawn from five diverse data sources spanning general instruction following, code generation, creative writing, and expert-level deep research, until no new failure modes are identified. We evaluate the consistency of the taxonomy by measuring agreement among independent human annotators, observing fair agreement overall ( 87% pairwise agreement and 0.64 average Cohen’s kappa ). Finally, to support scalable diagnosis, we propose automated rubric quality metrics and show that they align with human failure-mode annotations, achieving up to 0.925 F1.

## 1 Introduction

Rubrics have become a central component of recent large language model (LLM) benchmarks and training pipelines, providing rich, interpretable, and scalable evaluation signals. They are widely used in evaluating open-ended generation tasks—ranging from general instruction following ( He et al., 2025 ) to research planning ( Goel et al., 2025 ) , as well as professional applications such as healthcare, law, and finance—where fully verifiable ground truth is often unavailable ( Arora et al., 2025 ; Akyürek et al., 2025 ; Shi et al., 2026 ) . By specifying task-specific textual criteria and scoring model outputs with an LLM-as-a-judge, rubric-based evaluation can substantially improve the alignment between automated rewards and human judgments ( Sirdeshmukh et al., 2025 ) .

Despite their growing importance, principled evaluation of rubric quality remains largely unexplored. Considerable effort has been devoted to designing human annotation workflows ( Akyürek et al., 2025 ) and to automatically generating rubrics ( Viswanathan et al., 2025 ; Xie et al., 2025 ; Liu et al., 2026 ; Rezaei et al., 2025 ) , but the quality of the resulting rubrics is rarely assessed. Instead, rubric quality is typically inferred indirectly through downstream performance (e.g., reinforcement learning outcomes) or agreement between rubric-based LLM judges and human preferences. However, such downstream signals conflate rubric quality with other factors, including judge behavior and task formulation, making it difficult to isolate failures caused by the rubric itself. As a result, rubrics produced through heterogeneous pipelines cannot be meaningfully compared, and there is no principled way to characterize how a rubric itself fails .

We introduce RIFT , the RubrIc Failure mode Taxonomy , a taxonomy for diagnosing failures in evaluation rubrics. RIFT is derived through a grounded-theory process ( Glaser and Strauss, 1967b ) based on expert critiques of diverse rubrics drawn from both human-authored and automatically generated sources. Using five representative human-curated and synthetic rubric data sources spanning general instruction following, code generation, creative writing, and expert-level deep research, we identify eight recurring failure modes and organize them into three higher-level dimensions of rubric quality: reliability , content validity , and consequential validity . We also observe systematic differences between human-authored and synthetic rubrics, motivating human-in-the-loop rubric creation that combines broad synthetic coverage with expert refinement. Although the taxonomy is grounded in the rubric collections studied here, the grounded-theory-based construction workflow is general and can be readily applied to new domains and rubric generation procedures.

Beyond defining the taxonomy, we study whether these rubric failures can be identified consistently. We validate RIFT through expert annotation and inter-annotator agreement among three independent human annotators, and further develop automated diagnostics that approximate RIFT labels using a combination of LLM-based classification and agreement- and stability-based signals. These diagnostics enable scalable analysis of rubric quality and make RIFT practical for real-world rubric development and iteration workflows. Our contributions are summarized as follows: • Taxonomy development. We introduce RIFT, a rubric failure-mode taxonomy derived using grounded theory ( Glaser and Strauss, 1967b ) , which identifies eight failure modes organized into three high-level categories: Reliability , Content Validity , and Consequential Validity .

• Systematic empirical grounding. We construct and analyze a dataset of 85 diverse rubrics with 255 expert annotations, drawn from five representative benchmarks and covering both human-authored and automatically generated rubrics.

• Automated diagnostics. We develop scalable automatic signals for detecting RIFT failure modes and achieve high agreement with expert annotations, reaching up to 0.86 F1.

## 2 Related Work

Task-Specific Rubrics as Evaluation. Task-specific rubrics verify LLM performance by using weighted textual criteria, evaluated by an LLM-as-a-judge (LLMaJ) and aggregated into a single reward signal. Rubric-based evaluation improves agreement with human judgments ( Sirdeshmukh et al., 2025 ) and is widely used for benchmarks in unverifiable domains (e.g., healthcare, law, and finance) ( Arora et al., 2025 ; Akyürek et al., 2025 ; Shi et al., 2026 ) and open-ended tasks such as general-purpose instruction following ( He et al., 2025 ) and research planning ( Goel et al., 2025 ) .

Issues With LLM Evaluation Benchmarks. Recent studies show that LLM benchmarks suffer from unreliable outcome verification, including widely used benchmarks such as SWE-bench-Verified and τ \tau -bench ( Chowdhury et al., 2024 ; Yao et al., 2024 ; Zhu et al., 2025b ) . Beyond outcome verification, naive LLM-as-Judge evaluation can introduce systematic bias and overconfident estimates which requires calibrated reporting ( Lee et al., 2026 ) . Prior work also reports substantial evaluation variance ( Madaan et al., 2024 ) and other weaknesses in current benchmarking practices ( Reuel et al., 2024 ; Eriksson et al., 2025 ) , motivating the need for more reliable evaluation design.

Operationalization and Validity in Measurement. RIFT’s three failure categories draw on established measurement-theory concepts. Reliability Failures parallel the problem of operational definitions ( Bridgman, 1927 ; Stevens, 1935 ) : when rubric criteria lack precise operationalizations, different judges interpret them differently, producing inconsistent scores ( Breznau et al., 2022 ; Carpentras, 2024 ) . Content Validity Failures build on the psychometric notion of content validity—whether an instrument adequately represents the construct it measures ( Sireci, 1998 ) —applied here to rubric criteria that miss or misrepresent the evaluation target. Consequential Validity Failures draw on consequential validity ( Iliescu and Greiff, 2021 ) , capturing rubrics whose design enables gaming or low discrimination downstream.

Failure Mode Taxonomy. Constructing failure mode taxonomies via human annotation is a common approach for identifying challenges in LLM-based systems. Zhu et al. (2025a) synthesises human analyses of agent trajectories into a unified failure taxonomy. Ma et al. (2025) and Cemri et al. (2025) further adopt grounded theory ( Glaser and Strauss, 1967a ) to iteratively build failure taxonomies through multi-rater consensus.

Prior work on benchmark reliability has largely focused on verifiable, agentic settings, while rubric-based evaluation in open-ended domains still lacks principled methods for assessing rubric quality in isolation. We therefore focus on evaluating rubric design and composition across datasets, with a particular emphasis on identifying failure modes. To our knowledge, RIFT is the first failure-mode taxonomy specifically designed to uncover structural shortcomings in rubrics themselves.

## 3 The RubrIc Failure mode Taxonomy

We define the RubrIc Failure mode Taxonomy (RIFT) as a generic framework for identifying and classifying failure modes in evaluation rubrics (Tab. 1 ). To avoid imposing a top-down schema, RIFT is developed using grounded theory ( Glaser and Strauss, 1967b ) , in which failure modes are iteratively derived from expert annotations and open-ended feedback. The resulting taxonomy organizes rubric failures into three core categories: Reliability (consistency and reproducibility of judgments; cf. the literature on operational definitions ( Bridgman, 1927 ; Stevens, 1935 ) ), Content Validity (alignment between rubric criteria and the intended evaluation target ( Sireci, 1998 ) ), and Consequential Validity (downstream usefulness and discriminative power of the rubric ( Iliescu and Greiff, 2021 ) ).

Data Sources. To ground the taxonomy in a wide range of settings, we analyze 85 rubrics with 255 expert annotations drawn from five data sources, spanning both human-authored and automatically generated rubrics: • Human-crafted. AdvancedIF ( He et al., 2025 ) provides expert-authored evaluation criteria for complex instruction following in general conversational tasks, and ResearchRubrics ( Sharma et al., 2025 ) focuses on factual grounding and multi-step analytical reasoning for deep research agents.

• Synthetic. WildChecklists ( Viswanathan et al., 2025 ) derives fine-grained rubric requirements from observed failure patterns, OpenRubrics ( Liu et al., 2026 ) constructs rubrics via contrastive analysis of preference pairs, and AutoRubrics ( Xie et al., 2025 ) induces reward-oriented rubrics through symbolic search. Together, these datasets and generation frameworks cover a broad range of domains, including coding and creative writing.

Taxonomy Development with Grounded Theory. We develop RIFT through an iterative process inspired by Grounded Theory, following prior work on failure taxonomies for multi-agent LLM systems ( Cemri et al., 2025 ) and practitioner guide in ( Chun Tie et al., 2019 ) . The process spans four sequential annotation rounds and covers 85 rubrics in total. Ae use purposive sampling to select rubrics from AdvancedIF , ResearchRubrics , WildChecklists , OpenRubrics , and AutoRubrics . These sources were chosen to maximize variation in rubric origin (expert-written or auto-generated), format (checklists, principles, or narrative rubrics), length, and domain. Full details on data composition are provided in Appendix A .

Three experts develop the taxonomy iteratively using standard Grounded Theory practices ( Chun Tie et al., 2019 ) . We use open coding to derive candidate failure modes from open-ended rubric critiques, constant comparative analysis to compare new observations against existing categories, and memoing to record emerging failure modes, boundary cases, and revision decisions. Data collection and analysis proceed concurrently across rounds: each annotation round informs the next revision of the taxonomy. The process continues until saturation condition is reached, which we define as the point at which additional rubrics no longer reveal new failure modes or require substantive taxonomy revisions. We operationalize this process as follows:

1. Bootstrapping (first round only). Experts independently write open-ended rubric critiques for each rubric. These critiques are provided to GPT-5.2 to propose an initial candidate taxonomy.

2. Labeling and feedback. Experts label each rubric using the current taxonomy. They also document failure modes not covered by existing categories and note issues in the taxonomy itself, such as unclear definitions, overlapping categories, or structural inconsistencies.

3. Panel refinement. The three-expert panel reviews all labels, critiques, and feedback, then revises the taxonomy before the next annotation round.

4. Convergence and finalization. The process terminates when experts agree that no new failure modes are emerging and no further rubric-driven taxonomy revisions are needed, yielding the final taxonomy in Tab. 1 .

A full per-source, per-round provenance breakdown is provided in Appendix A .

Validation and Agreement Analysis. We assess theoretical saturation and consistency of RIFT through a final manual annotation round. We observe fair to substantial agreement across identified failure modes, with an average Cohen’s kappa (C- κ \kappa ) of 0.64, Krippendorff’s alpha (K- α \alpha ) of 0.60, and a pairwise agreement rate (PWA) of 87%. A per–failure-mode analysis shows that some categories are more difficult to annotate consistently. In particular, Misaligned or Rigid , which captures cases where a rubric evaluates the wrong objective or imposes overly strict requirements, has the lowest PWA, reflecting variation in annotators’ thresholds for misalignment and strictness. In contrast, annotators show high agreement on the Low Signal category, consistently identifying rubrics that fail to provide sufficient evaluation signal.

Correlation With Human Preferences. Among the five data sources, only OpenRubrics and AutoRubrics provide paired judge-human preference labels, so we restrict this analysis to those two sources. We use judge-human preference agreement as an external validation signal for RIFT and find that rubrics with more RIFT failure mode labels are significantly more likely to exhibit judge–human misalignment (Pearson’s r r = 0.162, p p = 0.0021; average count = 3.66 vs. 3.15, p < p< 1e-4; effective size = 20) failure modes for misaligned vs. aligned cases), providing quantitative evidence that RIFT captures rubric properties relevant to downstream human-aligned evaluation.

Human-Crafted vs Synthetic Comparison. Tab. 2 summarizes the prevalence of RIFT failure modes identified in human-crafted vs. synthetic rubrics. Human-crafted rubrics tend to be more reliable and consequentially valid , but are more often Misaligned or Rigid , reflecting overly strict or incorrect assumptions. Missing Criteria arises in both settings: synthetic rubrics are often encompassing without precision, which can fail to operationalize criteria, while human-crafted rubrics tend to be more specific but can still omit some requirements. These patterns motivate human-in-the-loop rubric creation pipelines that use synthetic generation for broad coverage, then rely on expert review to sharpen and correct misalignments.

## 4 Automated RIFT Evaluators

To enable scalable assessment of rubric quality under RIFT, we develop automated evaluators for each failure mode. We evaluate these signals on an independent test set of 50 rubrics (10 per source, stratified equally across all five data sources) sampled without replacement from rubrics not used during taxonomy development and manually annotated using the RIFT taxonomy. All 50 rubrics are evaluated by every automated signal; for both LLMaJ classifiers (GPT-5.2 and Gemini 3 Pro), each rubric is evaluated by both models. We implement several automated signals and examine how well they align with expert diagnostic labels: • LLM-as-Judge (LLMaJ). A rubric-conditioned failure-mode classifier trained using grounded-theory annotations and the RIFT taxonomy. We report results for two classifiers, GPT-5.2 and Gemini 3 Pro, and provide the taxonomy as part of the prompt (see Appendix C ).