RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics — part 3/4

Cleaned-text code points [31810, 47606). Bibliography omitted here; source.md includes it.

The prompt further specifies a detailed taxonomy philosophy and refinement guidelines that constrain the model’s proposed changes:

## Taxonomy Philosophy CRITICAL: This taxonomy will be used by human annotators. The primary goal is to create a taxonomy that is: • Compact : Aim for 7–10 total failure modes. Fewer distinct categories is ALWAYS better than many granular ones.

• Easily distinguishable : A human should be able to distinguish between any two failure modes in under 30 seconds. If two categories require careful reading to tell apart, they should be merged or their distinction should be clarified by refining the label names and or the description.

• Actionable : Each category must be clearly applicable without ambiguity.

Consolidation over proliferation: When in doubt, MERGE rather than add. Two failure modes that are 80% similar should become one category, not two. The cost of a slightly imperfect merge is far lower than the cost of a bloated, hard-to-use taxonomy.

## Guidelines

• Clear descriptions : Each failure mode description must be clear, specific, and actionable. The description should explicitly specify HOW to determine if a rubric exhibits this failure mode. An annotator should be able to read the description and confidently apply it to any rubric.

• No overlapping failure modes : The taxonomy should not contain failure modes with overlapping meanings. If two labels capture the same concept, merge them or refine them to make them distinct. Do NOT add a new failure mode if its meaning already exists under a different label.

• Self-contained rationales : Each rationale must be a self-contained justification that will be used for manual review. It should explain WHY this failure mode exists, what evidence from critiques supports it, and how it differs from other failure modes. A reviewer should understand the rationale without needing to see the original critiques.

• Cumulative applicability : The refined taxonomy must be applicable to ALL critiques that have been seen in this session (including previous batches), not just the current batch. Do not remove or change failure modes in ways that would make them inapplicable to earlier critiques that supported them.

## Task

Analyze BOTH the rubric critiques and taxonomy critiques above. Before adding any new failure modes, first consider whether existing categories should be merged.

FIRST: Consider merging existing failure modes when:

• Two or more categories have similar descriptions or capture closely related issues

• Categories are difficult to distinguish without careful reading

• A broader category could capture multiple narrower ones without losing important distinctions

• The taxonomy has grown beyond 12 failure modes

PREFERRED action - merge: Combine overlapping, redundant, or closely related labels into one. This is the most important refinement action. If you’re unsure whether two categories are distinct enough, merge them.

Add new failure modes ONLY when ALL of the following are true:

• The issue is clearly NOT capturable by ANY existing failure mode (even with minor rewording)

• The issue appears in MULTIPLE critiques (not just one annotation)

• The new category is easily distinguishable from ALL existing categories

• Adding it would NOT push the taxonomy beyond 12 failure modes

Other refinement actions:

• clarify: Make a label’s description clearer, more specific, or more actionable (especially clarifying HOW to identify the failure mode)

• split: Divide an overly broad label into more specific ones (use sparingly---only when a category is genuinely too broad to apply consistently)

• remove: Eliminate labels that are not useful, are duplicates, or are too similar to other categories

• rename: Change a label name to be more descriptive

Output: 1. failure_modes: The complete list of failure modes after applying changes. Each failure mode should have:

• label: concise identifier (e.g., contradictory_criteria, missing_edge_cases)

• description: clear, specific, and actionable description that explains HOW to determine if a rubric has this failure mode (what to look for, what conditions must be met)

• rationale: a self-contained justification for this failure mode that can be understood without seeing the original critiques. Explain why it exists, what patterns it captures, and how it differs from related failure modes. If this is a NEW category, explicitly explain why it cannot be captured by any existing category.

• examples: REQUIRED: 3-5 pass_examples AND 3-5 fail_examples for each failure mode. Multiple diverse examples are essential for annotator training. You may use real examples from the annotations or synthesize clear illustrative examples. Each example should illustrate a distinct scenario or nuance.

2. changes_summary: A list of strings describing what changes you made (e.g., "Added ‘contradictory_criteria’ based on rubric critiques", "Clarified description of ‘ambiguous_criterion’", "Merged ‘x’ and ‘y’ into ‘z’")

If no changes are needed based on these critiques, return the current running refinement unchanged with an empty changes_summary.

## Appendix D RIFT: RubrIc Failure Mode Taxonomy

This section provides the complete descriptions of each failure mode in the RIFT taxonomy (Tab. 1 ), including detailed decision rules for annotation. Each failure mode includes: (1) when to apply the label, (2) how to determine whether a rubric exhibits the failure mode, and (3) boundary conditions specifying when not to apply the label and which alternative label to consider instead. For each failure mode, we also provide illustrative pass examples (rubrics that do not exhibit the failure mode) and fail examples (rubrics that do exhibit the failure mode). Note: in the “Do NOT apply” cross-references below, CSV-internal label identifiers have been replaced with the display names used in Tab. 1 .

### D.1 Reliability Failures

Reliability failures lead to inconsistent grading across annotators or evaluation runs, reducing the reproducibility and trustworthiness of rubric-based evaluation.

#### Subjective.

Apply when the rubric uses inherently subjective evaluative terms (e.g., “clear,” “appropriate,” “credible,” “comprehensive,” “professional,” “engaging,” “well-written,” “good sources”) and does NOT sufficiently anchor them with objective expectations.

How to determine: • Identify criteria dominated by inherently subjective terms.

• Check whether the rubric provides ANY anchoring attempt such as: – concrete checklists (“includes X/Y/Z”),

– measurable thresholds (word count, required sections, required elements),

– examples/anti-examples of what qualifies vs does not qualify, or

– explicit decision rules (“count as clear if it defines the term and gives one example”).

• If the rubric relies primarily on grader judgment and provides no meaningful anchors, apply.

Important clarification: • Do NOT apply if the rubric gives examples or non-trivial decision rules that explain what the subjective term means (even if the term is still somewhat subjective).

Do NOT apply if: • The core issue is missing expected answers/tolerances or a bounded verification procedure for a groundable requirement (use Ungrounded ).

• The requirement is entirely absent (use Missing Criteria ).

Illustrative examples:

Input Context Rubric Pass Write a professional email declining a meeting. 2 pts: Includes a decline + proposes an alternative time. 2 pts: Uses a greeting and sign-off. 1 pt: No negative or insulting language. Fail Summarize the study. 10 pts: The summary is clear and sufficiently detailed.

#### Non-Atomic.

Apply when the rubric does not provide a parseable, consistently scorable structure OR uses bundled (non-atomic) criteria that prevent consistent partial credit.

Triggers (any sufficient):

Non-atomic (bundled) criteria • One scored item bundles multiple independently scorable requirements with no partial-credit rule or separable sub-scores (e.g., “clear, comprehensive, accurate, and well-cited” as a single 10-pt item).

Do NOT apply when: • Subparts are separately scored or the rubric provides explicit level anchors (e.g., “1 point each for A/B/C” or a 0–2 scale per dimension with definitions).

• The rubric is scorable but uses subjective language (use Subjective ) or is missing requirements (use Missing Criteria ).

Illustrative examples:

Input Context Rubric Pass Write a short answer with two supporting reasons. 2 pts: Answers the question. 1 pt: Reason #1 supports the answer. 1 pt: Reason #2 supports the answer. 1 pt: Total length <=150 words. Fail Summarize the article. 10 pts: Summary is clear, accurate, comprehensive, concise, and engaging.

#### Ungrounded.

Apply when the rubric requires verification that is plausibly groundable/boundable, but the rubric does not provide the necessary grounding (answer keys/acceptable variants/tolerances/decision rules) OR does not bound the verification procedure (what to check, how much to check, and how to judge conflicts).

How to determine (any sufficient): (A) Groundable determinate tasks lack grading anchors. The task has a knowable target output given fixed inputs (e.g., extraction, classification, translation, math, SQL result, code output), but the rubric provides no expected answers, acceptable variants, label mappings, tolerances, or decision rules. The criterion may be clearly worded (e.g., “totals are correct”), yet graders still lack what they need to check correctness.

(B) Open-world requirements lack bounded audit procedure. The rubric demands broad verification (e.g., “all facts are true,” “restaurants are open right now,” “fully original/no plagiarism,” “links work”) without bounding: what to check (scope/sample size), which sources/tools are allowed, how to resolve conflicting evidence, and the pass/fail threshold.

(C) Measurement standard is unspecified but could be made checkable. The rubric requires a measurement that depends on an unspecified standard (e.g., “exactly 20 pages”) without defining the rendering/formatting standard or offering a workable proxy.

Do NOT apply if: • The requirement is simply missing from the rubric (use Missing Criteria ).

• The main issue is subjective wording without anchors (use Subjective ). The rubric provides a representative list of examples to demonstrate expected content.

Illustrative examples:

Input Context Rubric Pass Extract all email addresses from the text. 1 pt per correct email address; accepted forms include plus-addressing. Gold list of emails: a@x.com, b.y@z.org, … Deduct 1 pt per missing email. Fail Compute the correct totals for these 30 invoices. 10 pts: Totals are correct.

### D.2 Content Validity Failures

Content validity failures arise when rubric criteria are misaligned with the intended evaluation target, either by grading the wrong objective or by failing to cover essential requirements.

#### Misaligned or Rigid.

Apply when the rubric (a) grades the wrong objective for the prompt or embeds incorrect assumptions, OR (b) imposes unnecessarily strict/narrow requirements not asked for by the prompt or reasonably inferred from the prompt, predictably penalizing prompt-faithful high-quality answers.

How to determine (any applies): • Wrong task / shifted objective: makes non-requested deliverables mandatory for points.

• Incorrect embedded assumptions: assumes a context not in the prompt (jurisdiction, audience, tools, constraints) and scores accordingly.

• Penalizes good practice: scores down reasonable caveats/uncertainty/safety practices when the prompt does not forbid them.

• Arbitrary brittleness/over-constraint: mandates a specific tool/library/method/structure/formatting or false precision when multiple reasonable alternatives would satisfy the prompt.

Do NOT apply when: • The prompt itself imposes the strictness at any point (e.g., exact JSON keys, or a direct instruction from the user earlier in a chat conversation).

• The requirement is missing entirely (use Missing Criteria ).

• The main problem is internal contradiction (use Self-Contradictory ).

• The main problem is rubric-level proxy gaming (use Hackable ).

Illustrative examples:

Input Context Rubric Pass Write Python code to parse CSV. 5 pts: Correct parsing. 3 pts: Handles quoted commas. 2 pts: Includes brief usage example. (Does not mandate pandas vs csv module.) Fail Write a haiku about winter. 5 pts: Includes at least 5 academic citations. 5 pts: Uses APA format reference list.

#### Missing Criteria.

Apply when the prompt implies at least one checkable must-have requirement, but the rubric provides no criterion that allows a grader to evaluate that requirement at all.

How to determine: 1. List the prompt’s core requirements: • required deliverables/components,

• must/must-not constraints,

• required format/ordering/sections,

• and genre-critical qualities the prompt clearly expects (e.g., functional correctness for code; “two sentences”; “valid JSON”; “include 10 items”; “chronological order”).

2. For each requirement, check whether ANY rubric criterion covers it.

3. If one or more requirements have no corresponding criterion, apply.

Do NOT apply if: • The rubric mentions the requirement but is vague or subjective (use Subjective ).

• The rubric mentions the requirement but it cannot be graded consistently due to missing keys/tolerances/bounded audit steps (use Ungrounded ).

• The rubric grades a different task or adds arbitrary constraints (use Misaligned or Rigid ).

Illustrative examples:

Input Context Rubric Pass Return ONLY valid JSON with keys: name (string) and age (integer). 3 pts: Output parses as JSON. 2 pts: Contains exactly keys name and age. 2 pts: name is a string; age is an integer. 1 pt: No surrounding commentary. Fail Write a 200-word email and include a subject line. 10 pts: Tone is professional. 5 pts: Grammar and spelling are correct.

### D.3 Consequential Validity Failures

Consequential validity failures reduce the downstream usefulness and discriminative power of rubric-based evaluation, even when individual criteria may be well-defined.

#### Hackable.

Apply when the rubric is gameable at the rubric level: a responder could easily achieve a top score by inflating proxy metrics (length, number of bullets/sections/items/citations/examples/brands, repeated keywords) without materially improving correctness, relevance, or fulfillment of the prompt—and the rubric lacks strong quality gates that tie points to substantive, prompt-aligned success.

Core question (required): • Could I easily achieve full marks on this rubric while still not satisfying the prompt requirements or producing a low-quality response?

How to determine (any sufficient): • Most points come from “more” ( ≥ \geq N tips/citations/examples/pros/cons/sections) while relevance, non-duplication, correctness, and prompt-specific success conditions are weakly specified or absent.

• Rewards merely asserting attributes (“quiet,” “fast Wi-Fi,” “no fees”) without requiring evidence, linkage to the task, or checks against duplication.

• Counting proxies dominate while key prompt requirements have only weak gates (e.g., no requirement that citations support specific claims; no requirement that items be distinct and on-topic).

Do NOT apply when: • Quantity minimums are paired with robust quality controls that make padding ineffective (e.g., each item must be non-duplicative, tied to a specific claim or user need, and verifiably grounded/bounded).

• The main issue is that the rubric is generic and doesn’t discriminate at all (use Low Signal ).

• The main issue is a specific criterion that shifts the task or overconstrains acceptable answers (use Misaligned or Rigid ).

Illustrative examples: