Reliable Chain-of-Thought via Prefix Consistency — part 8/8
Cleaned-text code points [109370, 118176). Bibliography omitted here; source.md includes it.
For the verbal-confidence calls used by CISC, Verbal binary, and P(True), the parser fails when the secondary completion does not yield an integer in the requested range (a 0–100 value for CISC, a 0/1 verdict for Verbal binary). P(True) reads the logprob of the binary call’s “1” token directly, so it is essentially never missing. Verbal failure rates show wide model spread, under 1 % 1\% for Nemotron2-9B and Ministral3-14B versus 19 % 19\% – 56 % 56\% for GPT-OSS and Nemotron3-30B on Verbal 0–100 (Table 22 ). The two analyses that consume verbal scores handle parse failures charitably to the verbal baselines: the AUROC table (Table 2 ) drops failures, while the WMV pipeline imputes the per-problem median (Appendix E ). Dropping the sample from the WMV pool would bias the per-problem answer distribution if failures correlate with the boxed answer, and would also hide the parser-failure rate from the evaluation. Keeping each failure at its natural pool frequency instead mirrors the deployment-time behavior in which a fresh draw would re-incur the same parse error rate. We therefore treat parser fragility as an intrinsic property of the baseline, with each failure contributing a neutrally-weighted vote. Table 23 reports AUROC ¯ \overline{\mathrm{AUROC}} under both modes side-by-side. The gap is at most 0.031 0.031 on every cell, so failure handling does not change the best signal on any cell. The secondary-call token cost is charged regardless of parse success, so a high failure rate raises the per-vote cost rather than reducing the effective sample count.
Abbreviations: FSci = FrontierScience-Olympiad, HMMT = HMMT Feb 2026, AIME = AIME 2025, Brumo = Brumo 2025.
##### Verbal 0–100 parser fragility.
The parser splits the model’s response on the first “)” and scans for any digit before it, expecting a near-immediate “ <digit>) ” completion (the prompt suffix already opens an explicit “ Proposed confidence: ( ”). The dominant failure pattern is a parenthetical or commentary token landing before the digit, for example “ score) 70 ”, “ analysis) 87 (commentary?) (final) 93 ”, “ rating)\n\n**70**. ”, “ (C) 80% ”, and “ value).\n(If you think...) ”. A smaller share of failures is completions with no digit at all, such as “ score). That is this answer: **safe**. ”. The model that does not fail at all, Nemotron2-9B, has a median successful response length of just 5 5 tokens, indicating that it emits “ <digit>) ” and stops. The failing models have median successful lengths of 30 30 – 200 200 tokens, meaning even their successful completions wander past the digit and leave many opportunities for parenthetical content to land before any number. This fragility illustrates a broader challenge of porting verbalized-confidence baselines, originally designed for instruction-following models that snap to a strict response template, to reasoning models that interleave thinking, commentary, and LaTeX before committing to a structured output. The original CISC and P(True) papers validated these methods only on instruction-tuned (non-reasoning) models that reliably follow the structured response template. We extend the same prompt and parser to reasoning models without modification, and a fully robust implementation in this regime would require redesigning both the elicitation prompt and the parser.
### H.2 Equivalence Judging
##### Judge configuration.
AIME 2025 is scored by exact match on the normalized answer (integer). For benchmarks where multiple surface forms can denote the same answer (HMMT Feb 2026, Brumo 2025, and FrontierScience-Olympiad), equivalence is determined by an LLM judge: the FrontierScience-Olympiad prompt follows the official grader released with the benchmark [ Wang et al., 2025 , Appendix B] , and the math prompt follows the format used in MathArena [ Balunović et al., 2025 ] and similar competition-math evaluations. Both prompts return a single-token “YES”/“NO” verdict which is mapped to an equivalence edge. Transitive closure over these edges yields canonical answer clusters for each problem.
Using the evaluated model as its own judge is a known limitation: a stronger external grader could shift absolute accuracies. We verify in Appendix D.6 that this does not change the relative comparisons reported here: re-scoring the pool with Claude Sonnet 4.6 on a 4-model, 3-benchmark subset (12 cells) preserves the sign of AUROC ¯ PC − AUROC ¯ best baseline \overline{\mathrm{AUROC}}_{\mathrm{PC}}-\overline{\mathrm{AUROC}}_{\mathrm{best\;baseline}} on all 12 cells (no cell flips), and the per-cell ordering of WMV methods is preserved at every operating point.
The full prompts used are:
##### FrontierScience-Olympiad.
You are grading an attempted answer to a science olympiad problem. You will be given the attempted answer and the reference answer. Evaluate strictly, but fairly. The reference answer is either a single number or expression in latex formatting, a chemical formula, a compound name, or a phrase referring to a specific name, entity, or method. Mark the attempted answer as correct if it fully matches the reference answer or is otherwise equivalent (e.g., an equivalent algebraic expression, a numerical number within 1 decimal place rounding of the reference answer (e.g., 6.69 ≈ \approx 6.7), an equivalent name for a compound/formula, equivalent when accounting for units, etc.). Mark it as incorrect if it is not equivalent to the reference answer. Attempted answer: {answer1} Reference answer: {answer2} Answer only ‘‘YES’’ if correct or ‘‘NO’’ if incorrect.
##### HMMT Feb 2026, Brumo 2025.
Determine whether two mathematical answers are numerically identical. Answer 1: {answer1} Answer 2: {answer2} Criteria: • If they represent exactly the same numerical value, respond ‘‘YES’’
• If they represent different values or one is not numerical, respond ‘‘NO’’
• Different formats (fractions, decimals, radicals, exponential notation) are acceptable if numerically equivalent
Examples: ‘‘1/2’’ and ‘‘0.5’’ → \to YES; ‘‘ 4 \sqrt{4} ’’ and ‘‘2’’ → \to YES; ‘‘ 2 3 2^{3} ’’ and ‘‘8’’ → \to YES; ‘‘3.14’’ and ‘‘ π \pi ’’ → \to NO; ‘‘x+1’’ and ‘‘1+x’’ → \to YES; ‘‘ x 2 x^{2} ’’ and ‘‘2x’’ → \to NO. Answer only ‘‘YES’’ or ‘‘NO’’.
## Appendix I Reproducibility Statement
##### Code and data.
The analysis pipeline and reproduction scripts are released at https://github.com/naoto-iwase/prefix-consistency , and the answer pool used to reproduce all reported numbers is released at https://doi.org/10.5281/zenodo.20082164 .
Models and datasets are publicly available (Table 24 ). Initial generation is stochastic (no vLLM seed), but aggregation and voting use seed 42 42 , so every reported number is bitwise reproducible from the pool.
##### Inference and sampling.
Each model is served through a vLLM OpenAI-compatible endpoint on 4 × 4{\times} NVIDIA A100 80GB GPUs, with context window 131,072 131{,}072 and a maximum output length of 100,000 100{,}000 tokens. Sampling uses each model’s recommended settings (Table 25 ).
Table 26 reports per-generation token counts at the default τ = 0.75 \tau{=}0.75 for all models, with the additional τ ∈ { 0.50 , 0.25 } \tau\in\{0.50,0.25\} values reported for GPT-OSS-20B (the model used in the τ \tau -sensitivity sweep of Appendix D.5 ).
##### Generation prompts.
For the three math benchmarks (HMMT Feb 2026, AIME 2025, Brumo 2025) the system prompt is “You are a helpful assistant specialized in solving mathematical problems.” For FrontierScience-Olympiad it is “You are an expert scientist solving olympiad-level problems in physics, chemistry, and biology.” Both append “Please reason step by step, and put your final answer within \boxed {} .” to each problem statement. Initial samples and the regenerated continuations from each truncation point share the same prompt.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .