LiveMathematicianBench: A Live Benchmark for Mathematician-Level Reasoning with Proof Sketches — part 4/4

Cleaned-text code points [46351, 56570). Bibliography omitted here; source.md includes it.

Figure 6 summarizes how the benchmark size changes across the main construction stages for each month. Full denotes the complete post-generation candidate pool, Quality Rubric > 5 >5 denotes the subset retained after the rubric-based quality filter, and Hard denotes the final released split after stem-nontrivial filtering and hardness calibration. The figure shows that the pipeline consistently reduces the candidate pool in a structured way rather than through ad hoc pruning: each month begins from a substantially larger generated set, is narrowed by quality control, and is then further distilled into a compact hard benchmark.

## Appendix E Evaluation Implementation Details

This section documents the released evaluation implementation in LiveMathematicianBench/eval/ . The current repository provides three backend-specific entry points: eval.py for Azure OpenAI models, eval_claude.py for Claude models accessed through the Anthropic API, and eval_vllm.py for models served through an OpenAI-compatible vLLM endpoint. Despite backend-specific client code and usage accounting, the three scripts share the same hard-set loader, prompt template, deterministic choice shuffling, answer parser, correctness rule, and output JSON structure.

#### Benchmark file format.

For each evaluation month, the scripts load the hard benchmark split from data/<month>/hard/qaEval_<month>_ge5_hard.json . Each JSON item contains a unique identifier and an mcq object with a question stem, one correct_choice , four distractors in choices , and auxiliary metadata such as mcq.meta.score . During evaluation, the released scripts consume the finalized mcq fields only; source theorem context, sketches, and other record-level annotations remain in the benchmark file but are not inserted into the released prompts.

#### Prompt construction.

Evaluation is implemented as a five-way multiple-choice selection task. For each item, the scripts combine the stored correct choice with the four distractors, then apply a deterministic shuffle using a per-item seed equal to the global seed plus the item’s dataset index. The shuffled options are relabeled as A – E . The model then receives a fixed system prompt instructing it to act as an expert mathematician, reason step by step, and place its final answer inside \boxed{} . The user prompt contains only two blocks: 1. the question stem from mcq.question ; and

2. the five relabeled answer choices.

The choices are formatted as “ (A) ... ”, “ (B) ... ”, and so on, separated by blank lines. All models were evaluated in the without-sketch setting. Due to budget constraints, we additionally ran sketch-aware evaluation only for the few models that performed best in the without-sketch setting.

#### Answer extraction and scoring.

After generation, the scripts recover the model’s predicted label by first searching for the last occurrence of \boxed{A} through \boxed{E} . If no boxed answer is found, the parser falls back to either a one-character response in A – E or, failing that, the last standalone capital letter in A – E . In the Azure and vLLM evaluators, answer extraction is applied directly to the returned response text. In the Claude evaluator, the Anthropic response is split into text and thinking blocks; the final prediction is extracted from the text block, while the thinking block is recorded separately. An item is marked correct iff the extracted label matches the post-shuffle label of the ground-truth option. Monthly accuracy is then computed as correct/total over the evaluated hard split.

#### Execution details.

All evaluators parallelize item-level inference with a ThreadPoolExecutor controlled by --concurrency . They also support repeated sampling through --n ; in that case each question is answered multiple times, a full per-sample list is stored under samples , and the top-level prediction fields intentionally mirror the first sample for backward compatibility. For every sample, the scripts record elapsed time, prompt tokens, completion tokens, total tokens, and reasoning-token counts whenever the backend exposes them.

The backend-specific command-line options differ as follows. The Azure evaluator requires --endpoint and --api-key , accepts --api-version , exposes both a client-wide --timeout and a per-sample --request-timeout , and optionally switches from chat completions to the Responses API through --use-responses-api ; this switch is also enabled automatically when the model name contains gpt-5.4 . The Claude evaluator requires --base-url , accepts --api-key , supports --thinking-budget to request extended thinking, and exposes --debug , --timeout , and --request-timeout . The vLLM evaluator requires --base-url , accepts an optional --api-key , exposes --temperature , --top-p , --debug , --timeout , and --request-timeout , and also supports optional --use-responses-api with automatic activation for model names containing gpt-5.4 .

Token accounting is normalized across backends. Azure records usage directly from the provider response. Claude and vLLM additionally normalize completion_tokens so that, when reasoning-token metadata is available, the reported completion count includes both visible output tokens and reasoning tokens, matching the OpenAI-style accounting used elsewhere in the benchmark.

#### Resuming and result serialization.

All scripts support fault-tolerant resumption through --resume . If an output file already exists, the evaluator loads the previous JSON, skips items whose prior record has a non- None model_answer and no recorded error, and re-runs unanswered or failed items only. Results are written to results/<month>/accuracy_test_<model>_<month>_<effort>.json , where model names are sanitized for filesystem safety and <effort> defaults to default when no reasoning level is specified. Each output file contains (i) test_info metadata such as model name, month, seed, maximum token budget, number of samples, and timestamps; (ii) duplicated aggregate accuracy summaries under summary.all and overall ; and (iii) detailed_results records containing the item identifier, extracted model_answer , shuffled correct_answer , raw_response , correctness flag, latency, token usage, reasoning_tokens , reasoning effort, n_samples , optional per-sample samples , auxiliary mcq.meta.score , and any backend error message. For Claude runs, each record additionally stores raw_thinking extracted from the Anthropic thinking block.

## Appendix F Additional Evaluation Results

### F.1 Sketch-Aware Accuracy with Substitution-Resistant Settings

Built upon the sketch-aware evaluation results in Figure , we further analyze how the presence of proof sketches interacts with and without substitution-resistant question formulations. We break down sketch-aware gains, separating standard multiple-choice questions from substitution-resistant items. Figure 7 shows that proof sketches improve performance in both settings, but the magnitude of the gain depends on the model and the choice style. GPT-5.4 benefits more strongly on the original-choice subset, whereas Gemini-3.1-pro-preview shows especially large gains on the substitution-resistant subset. This suggests that proof-level guidance can help models not only identify the correct theorem statement, but also reason through adversarial answer formulations designed to resist shortcut strategies.

### F.2 Cost Analysis

Accuracy alone does not capture how practical a model is for benchmark-scale evaluation. Figure 8 shows that the accuracy-cost frontier is led by GPT-5.4 rather than by the highest-token models. Gemini-3.1-pro-preview achieves the top accuracy at 43.5%, but it requires about 15.3k average completion tokens. By contrast, GPT-5.4 (high) reaches a similar 41.8% accuracy with only about 7.0k tokens, and GPT-5.4 (medium) remains close at 41.2% with about 3.8k tokens. In other words, GPT-5.4 combines high accuracy with materially lower token cost, while several other long-generation models, such as Qwen and Kimi, use far more tokens without matching GPT-5.4 or Gemini in accuracy.

At the same time, several open-weight or alternative frontier systems occupy distinct efficiency niches. Qwen3.5-397B-A17B and Kimi-K2.5 achieve mid-tier accuracy at higher token budgets, while GPT-oss-120b generates even more tokens without matching the top closed-model accuracy band. This pattern suggests that improvements in research-level mathematical reasoning cannot be reduced to simply allocating more inference budget: architectural and training differences still materially affect how efficiently models convert tokens into correct mathematical judgments.

## Appendix G Prompt Details

We provide below some examples of system prompts used in our construction pipeline. We will focus on prompts used in Stage 4 (Question-Answer Pair Generation) as it best demonstrates the methodological novelty of LiveMathematicianBench .

To show case our category-specific system prompts, we include prompts from two categories: biconditional/equivalence and implication as they are representative of our central design philosophy.

#### Stage 4a: Question Stem and Correct Choice Generation.

#### Stage 4b: Sketch-Adversarial Distractor Generation.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .