RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics — part 2/4

Cleaned-text code points [15884, 31810). Bibliography omitted here; source.md includes it.

• Majority-Vote LLMaJ (LLMaJ MV). The same rubric-conditioned classifiers run N = 5 N{=}5 times independently per rubric. Per-failure-mode labels are aggregated by strict majority vote: a label is retained only if at least ⌈ N / 2 ⌉ \lceil N/2\rceil runs predict it.

• Inter-rater reliability (IRR). An agreement-based signal computed as pairwise agreement (PWA) over rubric-conditioned preference labels produced by four preference labelers (GPT-5 mini, Claude Haiku 4.5, Gemini 3 Flash, and GPT-5.2.) The preferences are generated over all pairs of responses generated by six models (GPT-5 mini, GPT-5.2, Claude Haiku 4.5, Claude Sonnet 4.5, Gemini 3 Flash, and Gemini 3 Pro.)

• Alignment. An accuracy-based signal measuring how often weaker preference labelers (GPT-5 mini, Claude Haiku 4.5, and Gemini 3 Flash) agree with GPT-5.2 on the same rubric-conditioned response pairs.

• Reward variance. A stability-based signal defined as the variance of the aggregate rubric score produced by a rubric-conditioned LLMaJ (GPT-5.2) over four independent responses generated by GPT-5 mini for each test input.

Alignment with Expert Annotations. Results are reported in Tab. 3 . For each evaluator, failure-mode pair, we report F1 at the threshold that maximizes F1 on the test set. The single-run LLMaJs achieves moderate to good alignment on a subset of failure modes, Subjective and Misaligned or Rigid, but exhibits notable misalignment on Non-Atomic and Hackable. Aggregating N = 5 N{=}5 independent LLMaJ runs via majority vote (LLMaJ MV) typically improves performance on the eight failure modes over the single-run LLMaJ and becomes the best evaluator on four. Majority voting degrades Low Signal and Redundant Criteria, where the non-LLMaJ signals and single-run LLMaJ, respectively, remain the strongest evaluators. GPT-5.2 and Gemini 3 Pro have noticeably different strengths: Gemini 3 Pro is substantially better on Ungrounded (0.634 vs. GPT-5.2’s 0.444) and Subjective (0.909 vs. 0.861), but weaker on Missing Criteria (0.452 vs. 0.686) and Low Signal (0.400 vs. 0.667), indicating that different frontier models surface complementary failure modes. The non-LLMaJ-based methods complement the LLMaJ-based evaluators, and we expect further gains from prompt tuning, curating in-context examples, and combining majority voting with these complementary signals.

Grouping failure modes into reliability, content, and consequential categories highlights observed difficulties of automated rubric-quality evaluation. Reliability failures are identified more accurately, likely because they result in inconsistent rubric-based judgments that non-LLMaJ signals measure directly, and because RIFT definitions provide clear scoring guidance for LLMaJs. In contrast, content and consequential failures require detecting substance gaps or inaccuracies in rubric criteria and anticipating downstream effects.

Pairwise Model Agreement. Tab. 4 reports per failure mode and macro-averaged agreement between three frontier majority-vote LLMaJs (Sonnet 4.5, Gemini 3 Pro, GPT-5.2) on the 50-rubric RIFT test set. Overall cross-model agreement rates range from 0.730 0.730 to 0.812 0.812 and Cohen’s κ \kappa from 0.239 0.239 to 0.494 0.494 . Between Gemini 3 Pro and GPT-5.2, per failure mode, Cohen’s κ \kappa ranges from 0.0 0.0 to 0.573 0.573 . This suggests moderate taxonomy interpretation differences between current frontier models and motivates leveraging experts in the loop for RIFT evaluations on failure modes such as Consequential failure modes with low model agreement.

## 5 Conclusions

We introduced RIFT: a taxonomy for rubric failure modes derived from expert annotations of real evaluation rubrics and iterative refinement. A key takeaway from the RIFT development and annotation process is that synthetic rubrics often provide broad but imprecise coverage, while human-crafted rubrics are typically sharper but can be misaligned or overly rigid. Combining generation with targeted expert review can capture coverage while correcting misalignments.

We also implemented automated RIFT evaluators to detect and label failure modes, enabling scalable analysis. In practice, we envision RIFT supporting an iterative rubric improvement workflow, for instance: (1) diagnose : run automated evaluators to flag likely failure modes for a candidate rubric, (2) review : an expert inspects flagged failures using the taxonomy’s decision rules and confirms or dismisses each, (3) revise : apply targeted fixes guided by the failure mode (e.g., anchor subjective terms with concrete criteria, unbundle non-atomic items into separately scored sub-criteria, or add missing requirements), and (4) validate : re-run the automated signals to verify the revision resolved the flagged issues. We expect expert-in-the-loop workflows to remain important for assessing content and consequential validity. Experts can verify coverage, relevance, and missing criteria (with LLM critics to accelerate annotation ( McAleese et al., 2024 ) ), while consequential validity may require methods that directly measure a rubric’s impact in downstream use cases.

## 6 Limitations

RIFT is not intended to be an exhaustive taxonomy of rubric failure modes. The taxonomy is derived from the rubrics, artifacts, and annotation settings examined in this study, and additional failure modes may emerge in other domains, task formats, or stakeholder contexts. As a result, the current taxonomy should be viewed as a useful but incomplete characterization of rubric breakdown.

The generalizability of RIFT remains only partially validated. Our annotations were conducted on a bounded set of tasks and artifacts, and rubric quality may depend heavily on the application setting, evaluation goal, and stakeholder priorities. Future work should test the taxonomy across a broader range of domains, annotator populations, and use cases.

The annotation process itself is expensive and difficult to scale. Although expert annotation provides high-quality supervision, it is still subject to ambiguity and subjectivity, particularly for abstract rubric properties such as completeness or specificity. This limits both the scale of our study and the reliability with which fine-grained failure modes can be distinguished.

Finally, evaluator alignment with expert judgments still has substantial room for improvement. Agreement with expert labels is only an indirect proxy for practical utility, and future work should more directly measure how identifying different failure modes affects downstream outcomes such as rubric revision quality, evaluator robustness, and benchmark reliability.

## 7 Ethical Statement

Rubrics are often used in high-impact evaluation settings, so errors in rubric design can propagate into misleading judgments, poor model selection, or unfair downstream decisions. Our goal in introducing RIFT is to support more transparent and reliable rubric development by making common failure modes easier to identify and analyze. At the same time, we do not view automated rubric diagnosis as a substitute for human oversight, especially in sensitive or high-stakes applications. We believe expert review remains essential for ensuring that evaluation criteria are appropriate, fair, and aligned with the intended use case.

## Appendix A Data Source and Sampling Provenance

This appendix summarizes the data sources and sampling procedure used for taxonomy development (Section 3 ) and automated evaluator experiments (Section 4).

### A.1 Data source distribution.

We sample rubrics from five datasets selected to cover variation in rubric origin, format, length, and task domain. Tab. 5 summarizes each source.

### A.2 Sampling.

Rubrics are drawn by stratified uniform random sampling with equal counts per source, without replacement across rounds, and with a fixed random seed per round for reproducibility. The taxonomy development sample contains 85 rubrics across four rounds: Round 1 collects open-ended critiques for bootstrapping, while Rounds 2–4 use the evolving taxonomy for labeling. Each rubric is independently annotated by at least three experts. The test set is drawn independently with the same stratified procedure, contains 50 rubrics with no overlap with the taxonomy development sample, and is evaluated by all automated signals in Section 4. For the LLM-as-Judge classifiers, both GPT-5.2 and Gemini 3 Pro evaluate every test rubric. Tab. 6 summarizes both samples.

## Appendix B Additional Experiment Details

This appendix provides additional analyses for the automated RIFT evaluators. First, we test whether the Alignment and Reward Variance signals are sensitive to the choice of judge model by replacing GPT-5.2 with Gemini 3 Pro Preview. Second, we report threshold-free ROC-AUC for the non-LLMaJ evaluators to complement best-threshold F1 and better account for class imbalance in the failure-mode labels.

### B.1 Alignment and Reward Variance Judge Model Ablation

The main paper reports two automated signals whose “judge” role is instantiated with GPT-5.2. Alignment measures how often weaker preference labelers (GPT-5 mini, Claude Haiku 4.5, and Gemini 3 Flash) agree with a strong reference labeler (GPT-5.2) on the same rubric-conditioned response pairs. We ablate by swapping the reference labeler to Gemini 3 Pro Preview, holding the three weaker labelers and the response pairs fixed. Reward variance measures stability of the aggregate rubric score under a rubric-conditioned LLM-as-Judge (LLMaJ) that scores four independent responses from GPT-5 mini per test input; we ablate by swapping that judge from GPT-5.2 to Gemini 3 Pro Preview, holding the response generator and evaluation protocol fixed. As shown in Tab. 7 , both signals are largely judge-model robust on Subjective and Non-Atomic, with the largest swings on consequential failure modes (notably Hackable and Low Signal). We hypothesize that is caused by label imbalance in the evaluation dataset.

### B.2 Additional Analysis for non-LLMaJ RIFT Evaluators

We additionally report threshold-free ROC-AUC results for automated failure-mode detection to address the concern that F1 at the best threshold on the same evaluation set may overstate performance due to in-sample threshold tuning. In particular, some failure modes are highly imbalanced (e.g., only 4 positives for Hackable ), making threshold-dependent F1 especially unstable. We therefore compute direction-agnostic ROC-AUC as max ⁡ ( AUC , 1 − AUC ) \max(\mathrm{AUC},1-\mathrm{AUC}) , matching the direction-agnostic threshold sweep used for F1.

We evaluate all evaluators except LLMaJ , including IRR , Alignment , Reward variance . Across the 48 joined rubrics, IRR has mean 0.712 (std. 0.149), Alignment has mean 0.704 (std. 0.166), and Reward Variance has mean 0.025 (std. 0.053).

Tables 8 and 9 show that threshold-free AUC is generally more conservative than best-threshold F1. The clearest example is Subjective , where F1 reaches 0.854 but best AUC is only 0.640, suggesting that much of the F1 comes from class prevalence rather than strong ranking ability. In contrast, Hackable has the strongest AUC (0.807 with Alignment) despite a modest best F1 of 0.400, indicating unstable signals under severe class imbalance. The strongest diagnostic also differs by failure mode, i.e., Alignment works best for Hackable , Misaligned or Rigid , and Missing Criteria ; IRR works best for Non-Atomic ; and Reward Variance works best for Ungrounded , Low Signal , and Redundant Criteria , suggesting that the three signals provide complementary information. Finally, correlations with the total number of failure modes are weak ( r = 0.247 r=0.247 for IRR, r = 0.222 r=0.222 for Alignment, and r = 0.052 r=0.052 for Reward Variance), so these signals are more informative about which failure mode is present than how many are present overall.

## Appendix C Prompts

This section provides the prompt templates used in the RIFT pipeline. Template variables are denoted with double curly braces (e.g., {{variable}} ).

### C.1 LLM-as-a-Judge Annotation Prompt

The following prompt template is used by the automated LLM-as-a-Judge (LLMaJ) evaluator described in Section 4. For each rubric under evaluation, the prompt is populated with the complete RIFT taxonomy—including all failure mode descriptions and pass/fail examples—alongside the rubric’s input context and rubric text. The LLMaJ returns structured output: for each identified failure mode, the model provides the failure mode label, a justification for why the failure mode applies, and a direct quote from the rubric exhibiting the issue.

You are an expert at evaluating rubric quality. Analyze the following rubric against the failure mode taxonomy and identify any issues. The rubric is designed to evaluate the quality of an AI model’s response to a given prompt. ## Failure Mode Taxonomy [For each failure mode in the taxonomy:] ### {{failure_mode.label}} Description: {{failure_mode.description}} **Pass Examples** (rubric does NOT exhibit this failure mode): [For each pass example:] - Input: {{example.input_context[:150]}}... Rubric: {{example.rubric[:200]}}... **Fail Examples** (rubric DOES exhibit this failure mode): [For each fail example:] - Input: {{example.input_context[:150]}}... Rubric: {{example.rubric[:200]}}... [End of taxonomy loop. If no failure modes are defined:] No failure modes defined yet - suggest any issues you observe. ## Input Context {{input_context}} ## Rubric to Evaluate {{rubric}} ## Task Identify which failure modes from the taxonomy apply to this rubric (if any).

The LLMaJ is configured to return a structured JSON response conforming to the following schema:

{ "suggested_labels": [ { "label": "<failure mode label from taxonomy>", "justification": "<why this failure mode applies>", "quote": "<specific rubric quote exhibiting the issue>" }, ... ] }

### C.2 Taxonomy Refinement Prompt

The following prompt template drives the iterative taxonomy refinement process described in Section 3 (Tab. 1 ). During each iteration of the grounded theory pipeline, expert annotator feedback—comprising open-ended rubric critiques and taxonomy critiques—is batched and provided to GPT-5.2 alongside the current taxonomy state. The model proposes refinements (merges, additions, clarifications, splits, removals, or renames), which are then reviewed and finalized by the expert panel.

You are an expert at analyzing rubric quality feedback and refining failure mode taxonomies. Your task is to output a complete refined failure mode taxonomy. ## Original Failure Mode Taxonomy This is the original taxonomy before any refinements in this session: [For each failure mode: label and description. If none defined: “No failure modes have been defined yet.”] ## Current Running Refinement This is the taxonomy as refined so far in this session (may be identical to original if this is the first batch): [For each failure mode: label, description, rationale, and counts of pass/fail examples. If none refined: “No refinements have been made yet.”] ## Annotator Feedback to Analyze Below are annotations with two types of critiques: - Rubric Critique: Issues the annotator observed in the rubric that were NOT captured by the original taxonomy labels (may suggest new failure modes) - Taxonomy Critique: Critique of the ORIGINAL taxonomy (unclear definitions, overlapping categories, missing categories, etc.). Note: these critiques were written against the original taxonomy, not the running refinement. [For each annotation in the batch:] Annotation {{loop.index}} Input Context: {{item.input_context}} Rubric: {{item.rubric}} Rubric Critique: (issues not captured by original taxonomy) {{item.rubric_critique or "None provided"}} Taxonomy Critique: (critique of the original taxonomy) {{item.failure_mode_critique or "None provided"}} [End of annotation loop]