The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning — part 4/5

Cleaned-text code points [47129, 63098). Bibliography omitted here; source.md includes it.

Math Coding Math AMC AIME Minerva Olymp. Avg. LeetC LiveC Avg. Qwen2.5-7b 43.8 31.3 15.6 14.7 19.0 24.9 26.1 18.4 22.3 Trained using 60K labeled prompts w/ SFT 48.2 30.2 10.0 17.6 22.4 25.7 18.3 18.3 18.3 w/ RLOO 73.0 57.8 23.3 31.2 34.2 43.9 28.3 26.7 27.5 w/ GRPO 71.8 56.6 21.1 25.0 35.9 42.1 25.0 25.8 25.4 Our unsupervised methods trained on 60K unlabeled prompts EM-FT w KL ( N = 4 N=4 ) 66.4 53.0 21.1 33.5 34.5 41.7 26.7 16.8 21.8 EM-FT w KL ( N = 1 N=1 ) 65.6 54.2 16.7 26.8 34.8 39.6 28.3 13.7 21.0 EM-FT w/o KL ( N = 1 N=1 ) 67.2 51.8 14.4 33.1 34.4 40.2 28.3 17.2 22.8

## Appendix D Experimental Setup

### D.1 Evaluation Benchmarks

#### D.1.1 Math Reasoning

We adopted the data and evaluation pipeline from PRIME-RL codebase [ 13 ] for all math reasoning evaluations. The final answer of LM is the string inside the last \ b ​ o ​ x ​ e ​ d ​ { } \backslash boxed\{\} . All math benchmarks is evaluated on Pass@1 accuracy. Here, we will introduce each math benchmarks in detail.

##### MATH 500

MATH 500 contains 500 prompts sampled from the 5,000 test problems in the original MATH dataset [ 29 ] . The test set of MATH comprises challenging competition-style math problems drawn from sources such as the AMC and AIME exams. These problems span seven subjects—Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus—and are labeled with difficulty levels from 1 (easiest) to 5 (hardest). Each test problem includes a final boxed answer and a full step-by-step solution in LaTeX, enabling exact match evaluation and providing interpretable reasoning paths for model analysis.

##### AMC

We use the validation set from AI-MO’s AMC dataset [ 43 ] , which contains 83 problems extracted from AMC 12 (2022 and 2023). The AMC dataset comprises math competition problems from the American Mathematics Competitions (AMC), covering a broad range of topics including arithmetic, geometry, combinatorics, and algebra. It is designed to evaluate a language model’s capacity for mathematical reasoning, abstraction, symbolic manipulation, and logical deduction.

##### AIME

We use the validation set from AI-MO’s AIME dataset [ 43 ] , which includes 90 problems drawn from AIME 22, AIME 23, and AIME 24. The AIME (American Invitational Mathematics Examination) dataset features high-difficulty problems intended for top-performing students from the AMC competitions. These problems typically require advanced mathematical reasoning across topics such as algebraic manipulation, number theory, combinatorics, and geometry. Each problem involves multiple intermediate steps and provides minimal guidance, making the dataset well-suited for evaluating a language model’s ability to perform multi-step symbolic reasoning and maintain logical coherence throughout extended problem-solving chains.

##### Minerva Math

Minerva Math refers to the OCWCourses evaluation dataset introduced in the Minerva paper [ 42 ] . This dataset comprises 272 undergraduate-level STEM problems sourced from MIT OpenCourseWare (OCW), covering topics such as solid-state chemistry, information theory and entropy, differential equations, and special relativity. Each problem is self-contained and accompanied by a clearly defined final answer—either numeric (191 problems) or symbolic (81 problems)—enabling automatic evaluation. The dataset is designed to assess a language model’s ability to perform multi-step scientific reasoning, including the application of domain-specific knowledge, symbolic manipulation, and precise quantitative computation.

##### OlympiadBench

OlympiadBench [ 28 ] is a large-scale bilingual multimodal benchmark comprising 8,476 Olympiad-level mathematics and physics problems sourced from international competitions such as the IMO and IPhO, as well as elite Chinese contests and Gaokao mock exams. The dataset evaluates LLMs on advanced mathematical and physical reasoning tasks, including multi-step logical deduction, symbolic manipulation, equation solving, theorem proving, and visual-spatial understanding from diagrams. Its high difficulty, multimodal diversity, and expert-level content make it a rigorous benchmark for assessing scientific reasoning capabilities. In this work, we use the open-ended, English, text-only subset of math competition problems as our evaluation set, which contains a total of 674 problems.

#### D.1.2 Coding

##### LeetCode

LeetCode benchmark is introduced in the technical report of DeepSeek-Coder [ 26 ] . This benchmark consists of 180 recent competition-level programming problems from LeetCode, collected between July 2023 and January 2024. Each problem is paired with 100 test cases to ensure comprehensive coverage, and prompts are constructed in a standardized format to simulate realistic coding challenges. This dataset evaluates a language model’s ability to perform code generation in competitive programming settings, testing a wide range of capabilities such as problem comprehension, algorithm design, logic reasoning, and implementation precision. The inclusion of varying difficulty levels (Easy, Medium, Hard) allows for a thorough assessment of a model’s generalization and problem-solving skills under pressure. During evaluation, the python code will be extracted from model’s response and run the test cases. Performance is reported using the Pass@1 metric, indicating the proportion of problems for which the model’s first generated solution passes all test cases.

##### LiveCodeBench

LiveCodeBench [ 34 ] is a large-scale, contamination-free benchmark designed to holistically evaluate the code-related capabilities of large language models. It comprises 612 recent coding problems collected from competitive programming platforms such as LeetCode, AtCoder, and Codeforces between May 2023 and August 2024. Unlike traditional benchmarks that focus solely on code generation, LiveCodeBench assesses four key skills: (1) code generation from natural language, (2) self-repair using error feedback, (3) code execution via output prediction, and (4) test output prediction without implementation. This comprehensive setup evaluates not only functional correctness but also a model’s ability to debug, reason about program behavior, and construct tests. Its time-segmented evaluation protocol further supports rigorous detection of training data contamination. In this work, we adopt the V2 release, which includes 511 problems released between May 2023 and May 2024. We evaluate methods on the code generation ability. The model is given a natural language problem description and tasked with generating a correct program. Correctness is evaluated by passing all test cases. The Pass@1 metric is used, measuring the fraction of problems where the first generated solution passes all tests.

#### D.1.3 Scientific Reasoning

##### UGPhysics

UGPhysics [ 88 ] is a large-scale bilingual benchmark designed to evaluate undergraduate-level physics reasoning in large language models. It comprises 5,520 problems (11,040 including both English and Chinese versions) spanning 13 physics subjects and 59 topics, featuring diverse answer formats such as numerical values, symbolic expressions, equations, and intervals. Each problem is annotated with one of four reasoning skill types: Knowledge Recall, Laws Application, Math Derivation, and Practical Application. Unlike prior benchmarks that primarily target multiple-choice or high-school-level content, UGPhysics emphasizes deeper problem-solving and derivation skills, requiring LLMs to perform symbolic reasoning, apply physical laws, and handle real-world quantities. This dataset is designed to assess LLMs’ ability to carry out multi-step physics reasoning beyond simple mathematical manipulation. In this work, we evaluate on the Semiconductor Physics subset, which consists of 372 problems. During evaluation, models answer physics problems using prompts tailored to different answer types for improved extraction and judgment. A rule-based correctness assessment then extracts answers from the model’s output and reports accuracy.

##### SciCode

SciCode [ 78 ] is a challenging, scientist-curated benchmark designed to evaluate large language models (LLMs) on real-world scientific coding tasks across 16 subfields in mathematics, physics, chemistry, biology, and materials science. It comprises 80 complex main problems, each decomposed into multiple subproblems (338 in total), requiring models to integrate domain-specific scientific knowledge, reasoning, and Python programming. Each problem includes gold-standard solutions and test cases, with optional expert-authored background context. SciCode assesses LLMs’ abilities in multi-step problem decomposition, code synthesis from scientific instructions, and integration of partial solutions. By emphasizing scientific reasoning and simulating real-world workflows, it provides a rigorous benchmark for evaluating LLMs as scientific assistants. In our paper, we evaluate methods on the test set, which contains 65 main problems and 288 subproblems in total. Models generate Python functions for each subproblem and integrate them into a complete solution. Evaluation is fully automatic, using gold test cases for both subproblems and main problems. The evaluation is conducted with and without scientific background information, as well as using gold or generated prior solutions, to measure realistic performance. Success requires all subproblems and the main solution to be correct. Final accuracy is reported as the percentage of problems achieving success.

#### D.1.4 Individualistic Value Reasoning

##### IndVal

The IndVal dataset [ 36 ] is derived from the World Values Survey (WVS) and consists of natural language statements expressing individual human values and preferences from over 93,000 participants worldwide. Each participant responds to 253 questions grouped into 13 distinct categories, including Social Values, Economic Values, and Science & Technology, among others. The dataset standardizes these diverse survey responses into over 900 value-expressing statements per individual, enabling evaluation of language models’ ability to reason about individualistic human values in a bottom-up, granular manner. This dataset is particularly valuable as it challenges models to understand and predict nuanced personal value judgments beyond demographic stereotypes. Consequently, it reveals limitations in current language models’ capacity for individual value reasoning and promotes more equitable, personalized AI alignment that respects individual diversity. In our work, we randomly sampled 200 individuals excluded from training, uniformly distributed across all continents. For each individual, we randomly selected 4 statements per question category to form an individualistic value profile consisting of 52 known statements. Then, for each question category, we selected 2 statements not included in the known profile and asked the language model to select the most appropriate statement from the given candidates based on the individual profile. This process resulted in an evaluation dataset containing 5,400 prompts. During evaluation, the selected statement is extracted from the model generation and compared with the ground truth. Final performance is reported as accuracy.

### D.2 Training Data

##### Math Reasoning

We randomly sampled 35K prompts from Numina math training set [ 43 ] as our math training prompts. The NuminaMath dataset contains 860,000 problems ranging from high-school to advanced competition levels, including Olympiad, AMC, AIME, and Chinese K-12 math problems.

##### Coding

We randomly sampled 25K prompts from Eurus-2-RL training dataset’s coding split [ 13 ] as our coding training prompts. The Eurus-2-RL training dataset is a high-quality reinforcement learning dataset consisting of about 457,000 math problems and 27,000 coding problems with outcome verifiers such as LaTeX answers for math and test cases for coding. The math problems are sourced from the NuminaMath-CoT dataset, which covers a wide range from Chinese high school math to International Mathematical Olympiad-level questions. The dataset is carefully cleaned, filtered, and reformatted into a direct question-answer format to ensure high quality and solvability. It is especially useful for training because it provides diverse, verified, and challenging problems covering different level of difficulties and tasks.

##### Individualistic Value Reasoning

The training dataset is derived from the IndVal dataset [ 36 ] . We sampled 900 individuals uniformly across seven continents. For each individual, we randomly selected 4 statements per question category to form an individualistic value profile comprising 52 known statements. Then, we sampled 6 questions per question category as queries, resulting in a total of 70,200 training prompts.

### D.3 Models

##### Qwen2.5 Family

For the entropy minimization for post-training experiments, we use two models derived from the Qwen2.5 family: Qwen2.5-Math-7B [ 89 ] and Eurus-2-7B-SFT [ 13 ] . Qwen2.5-Math-7B is selected as the base model for math reasoning tasks. We apply various post-training methods using the math training dataset and evaluate their performance on math reasoning benchmarks. This model serves as an excellent starting point for our experiments due to its strong capability in tackling complex mathematical reasoning problems. Eurus-2-7B-SFT is a SFT checkpoint obtained by further fine-tuning Qwen2.5-Math-7B on the Eurus-2-SFT-Data from PRIME-RL [ 13 ] , which is an action-centric chain-of-thought reasoning dataset. Since Eurus-2-7B-SFT further enhances the reasoning abilities of Qwen2.5-Math-7B, we select it for coding experiments, applying different post-training methods with the coding training dataset and evaluating on coding benchmarks.

For entropy minimization during inference time experiments, we select Qwen2.5-Math-7B-Instruct and Qwen2.5-7B-Instruct as the base models for math reasoning, coding, and scientific reasoning tasks because of their inherent instruction-following and reasoning abilities. Specifically, for SciCode, we include a more powerful model, Qwen2.5-32B-Instruct, due to its superior capability in handling complex problems, allowing us to evaluate the performance of EM-INF on stronger models.

##### Llama-3.1 Family

Llama-3.1 [ 24 ] is the second model family included in this work to ensure the generalizability of our methods across different models. For both the entropy minimization post-training and inference-time experiments, we select Llama-3.1-8B-Instruct as the base model.

### D.4 FLOPs Computation

Following [ 63 , 37 ] , we estimate the training FLOPs as 6 ​ P ​ D 6PD and inference FLOPs as 2 ​ P ​ D 2PD where P P is the number of parameters in the model and D D is the number of tokens. We use early stopping based on the validation set accuracy for all methods, which significantly changes the FLOPs utilized. • EM-FT: On each training step, we sample one trajectory from the model costing 2 ​ P ​ n 2Pn where n n is the sequence length followed by training costing 6 ​ P ​ n 6Pn . The total cost per step is thus 8 ​ P ​ n 8Pn . We set sequence length to 4096 4096 , batch size to 512 512 and the training early stops at 40 40 gradient steps costing 1.01 × 10 17 1.01\times 10^{17} FLOPs for a 7B parameter model.

• EM-RL, GRPO, RLOO: None of the methods use a reward model. We use GRPO and RLOO with output verification only. We sample N = 4 N=4 trajectories costing 8 ​ P ​ n 8Pn , then we make a forward pass on the reference model costing another 8 ​ P ​ n 8Pn followed by training costing 24 ​ P ​ n 24Pn . The total training cost per step is thus 40 ​ P ​ n 40Pn . We set sequence length to 4096 4096 , batch size to 512 512 and the training early stops at 100 100 gradient steps costing 13 × 10 17 13\times 10^{17} FLOPs for a 7B parameter model.