RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics — part 4/4

Cleaned-text code points [47606, 55827). Bibliography omitted here; source.md includes it.

Input Context Rubric Pass Provide 5 study tips. 1 pt each for 5 tips that are (a) non-duplicative and (b) each includes a concrete example of how to apply it. Fail Provide a recommendation. 5 pts: At least 10 pros. 5 pts: At least 10 cons. (No check for relevance or duplication.)

#### Low Signal.

Apply when the rubric as a whole does not discriminate candidate responses well for this prompt—i.e., it would give similar (often high) scores to many substantively different-quality responses—because the criteria are all generic, conditionally irrelevant, or too easy.

How to determine (rubric-level discrimination test): • Imagine 3–5 candidate responses ranging from weak to excellent.

• Ask: Would the rubric’s criteria/weights produce nearly equivalent different scores across them based on prompt-relevant success?

• If the rubric would likely award similar scores because most criteria are low-signal (e.g., “helpful,” “nice formatting,” “completed the task”) and there are few/no strong quality gates tied to the prompt’s real success conditions, apply.

Common signals: • Most points are allocated to generic writing quality/tone/formatting that is not central for this prompt.

• Criteria are trivially satisfied by any minimally on-task response (e.g., “schedule exists”).

• The rubric could be pasted into many unrelated tasks with little or no change.

Do NOT apply when: • The rubric is instead missing prompt-imposed must-haves (use Missing Criteria ).

• The rubric imposes the wrong constraints/assumptions (use Misaligned or Rigid ).

• The rubric is gameable specifically via quantity/proxies (use Hackable ).

Illustrative examples:

Input Context Rubric Pass Return JSON only with required keys. 6 pts: Valid JSON with required keys. 4 pts: No extra text outside JSON. Fail Return only a SQL query. 5 pts: Response is helpful. 5 pts: Uses appropriate tone.

#### Redundant Criteria.

Apply when two or more rubric criteria substantially evaluate the same underlying requirement such that the same behavior is rewarded/penalized multiple times.

How to determine: • Additional-signal test (primary): If these criteria are separate, do you get genuinely different evaluation signal, or are you just re-awarding the same property?

• Remove-one test: If removing a criterion would not meaningfully change what is evaluated (only point allocation), it is redundant.

• Includes near-duplicates and cases where one criterion fully subsumes another.

Important clarification (do NOT apply for mere dependencies): • Do NOT apply just because criteria are related or one tends to enable another.

• If Criterion B is a prerequisite/necessary condition for Criterion A (or vice versa) but still measures a distinct dimension (e.g., “valid JSON” and “has required keys”; “code compiles” and “passes tests”), that is NOT redundancy.

Do NOT apply when criteria are related but clearly distinct checks (e.g., factual accuracy vs clarity; format compliance vs correctness; presence of citations vs whether citations support claims).

Illustrative examples:

Input Context Rubric Pass Write a research summary with citations. 3 pts: Claims are supported by citations. 2 pts: Writing is well-organized. 2 pts: Includes limitations of the evidence. Fail Essay rubric. 5 pts: Clear writing. 5 pts: Clarity of prose. 5 pts: Writing is easy to understand.

## Appendix E Qualitative Analysis of Failure of LLMaJ in Detecting Failure Modes

To understand where the automated annotator disagrees with humans, we examine two difficult failure modes for LLM judges: Hackable or Proxy-Based Scoring and Ungrounded Verification . On the evaluation subset, these categories have F1 scores of 0.444 and 0.545, respectively. Both require reasoning beyond surface-level rubric critique: Hackable asks whether a responder could exploit the rubric, while Ungrounded asks whether a grader could actually verify the rubric.

#### Hackable : rewarding presentation instead of task success.

A representative false negative comes from a JSON-to-SQLite coding task, where the user asks how to save contacts from a JSON file into SQLite. Humans label the rubric as Hackable , but the automated annotator does not.

The rubric rewards presentation features such as step-by-step guidance, contextual explanation, and beginner-friendly writing. However, it does not require working code or any check that the JSON is actually parsed and saved into SQLite. A response could therefore look polished and detailed while still containing broken or incomplete code. The automated annotator catches nearby issues such as subjectivity and missing criteria, but misses the core problem: the rubric can be optimized by sounding competent rather than solving the task.

To test whether this is a capability issue or a prompting issue, we re-prompted the model on the same example. Under the standard annotation prompt with explicit rejection reasoning, the model still rejected Hackable :

This shows that the model interprets Hackable too narrowly. It looks for explicit numeric proxies, such as the number of bullets or citations, and misses softer proxies such as verbosity, structure, and tutorial-style explanation.

We then used a focused adversarial prompt that asked the model to imagine a responder trying to game the rubric. With this prompt, the model reversed its verdict:

This result suggests that the model can detect quality-proxy gameability when explicitly prompted to reason adversarially. The standard prompt does not reliably trigger that reasoning. In this case, the model treats hackability as a surface pattern to match, rather than asking whether the rubric can actually be exploited.

#### Ungrounded : clear requirement, missing verification procedure.

A representative false negative for Ungrounded Verification comes from a C# deadlock task. The model sees that the rubric is weak, but misses that the main requirement is not verifiable from the rubric alone.

The central criterion—“avoids the deadlock issue”—is clear at a high level, but the rubric does not say how to verify it. It gives no test case, expected behavior, execution setup, or pass/fail rule. A grader would need outside knowledge of C# async behavior and would have to decide independently whether the proposed solution works. This is an Ungrounded Verification problem: the criterion exists, but the rubric does not provide enough information to check it consistently.

Under the standard annotation prompt, the model focuses on general rubric weakness instead:

This explains why the model assigns nearby labels such as Subjective , Non-Atomic , and Trivial . However, it misses the main issue: the model never asks what evidence a grader would need to verify that the solution avoids deadlock. It does not check whether the rubric provides a test case, expected behavior, execution condition, or decision rule.

This is the main failure pattern for Ungrounded Verification : the model notices that the rubric is weak, but maps the problem to surface-level labels instead of the missing verification procedure. A better annotator needs to explicitly ask, for each factual or technical criterion, what the grader should check, how they should check it, and what counts as passing.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .