The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning β€” part 3/5

Cleaned-text code points [31174, 47129). Bibliography omitted here; source.md includes it.

Baseline: We use adaptive temperature scaling to reduce the entropy of the models’ output distribution. Starting from logits 𝒛 t {\bm{z}}_{t} we gradually decrease the temperature Ο„ \tau in the softmax until the entropy reaches a target value. We search for the largest Ο„ > 0 \tau>0 such that β„‹ ⁑ ( Οƒ ⁑ ( 𝒛 t / Ο„ ) ) ≀ max ⁑ ( Ξ± ​ β„‹ ​ ( Οƒ ⁑ ( 𝒛 t ) ) , Ξ΄ ) \mathcal{H}(\sigma({\bm{z}}_{t}/\tau))\leq\max(\alpha\mathcal{H}(\sigma({\bm{z}}_{t})),\delta) , where Ξ± ∈ ( 0 , 1 ) \alpha\in(0,1) controls the fraction of entropy to reduce. We find Ξ± ∈ ( 0.4 , 0.6 ) \alpha\in(0.4,0.6) works the best.

###### Proposition 1 .

(Informal) Both adaptive temperature scaling and logit optimization reduce the entropy of a model’s output distribution. However, in high-uncertainty settings (i.e., when the model’s predictive distribution has high entropy), logit optimization can change the order of non-top logits. In contrast, temperature scaling preserves the order of logits and merely sharpens or flattens the distribution proportionally. Both temperature scaling and logit optimization will keep the token with the top logit and increase its probability, making them less likely to change the model’s behavior when it is highly confident (Appendix A ).

Sub w/ BG Main w/ BG GPT-4o 35.4 9.2 GPT-4-Turbo-2024-04-09 33.7 0 9.2 Claude3.5-Sonnet 35.4 12.3 Claude3-Opus 26.7 0 4.7 Claude3-Sonnet 25.7 0 4.7 Gemini 1.5 Pro 30.6 0 7.7 Qwen2.5-7b-Instruct 11.5 0 0.0 Greedy decoding ( t = 0 t=0 ) 13.4 0 3.0 Iterative-refinement ( n = 2 n=2 ) 12.1 0 0.0 Adaptive Temp 13.2 0 0.0 EM-INF 16.7 0 1.5 Qwen2.5-32b-Instruct 25.4 0 4.6 Greedy decoding ( t = 0 t=0 ) 25.4 0 4.6 Iterative-refinement ( n = 2 n=2 ) 24.7 0 7.6 Adaptive Temp 28.8 0 7.6 EM-INF 27.1 10.7

EM-INF for scientific coding: We apply EM-INF to scientific coding in Table 1 . SciCode is a challenging code generation benchmark requiring significant domain knowledge and reasoning capabilities where even frontier models such as GPT4o struggle [ 78 ] . We observe that test time entropy minimization ( EM-INF , adaptive temperature) can improve performance of the base models consistently on easier subproblems and eventually on the more difficult main problem by 2.5 % 2.5\% . We note that logit optimization (EM-INF) is better than adaptive temperature scaling on SciCode by 3 % 3\% . Owing to the difficulty of SciCode, the models’ output distribution has large entropy. Under this scenario, EM-INF can change the relative order of the logits which likely leads to better reasoning chains. Whereas, adaptive temperature scaling will never change the relative ordering of the logits on each generation step – it only makes the distribution peakier (Proposition 1 ). Next, we observe that scaling through iterative-refinement [ 47 ] leads to some improvements over the base model, however, it is bottlenecked by the max context length of the model making it harder to refine long chain of thoughts. Whereas, EM-INF does not require refinement and maintains the same computational complexity as the base model when used with regular decoding.

We note that EM-INF with Qwen2.5-32B-Instruct can outperform several frontier models on the main problem without requiring refinement or editing model parameters by 3 % 3\% and over the base model by 6 % 6\% . Building on these observations, we put forward EM-INF as a practical, compute efficient inference time scaling method that does not require any supervision and makes no assumption on the nature of the problem.

EM-INF for math and coding: Table 3 compares the performance of EM-INF for test time scaling. We observe that minimizing the entropy of the model at test time without updating the parameters (EM-INF, adaptive temperature) improves the performance of the base model for almost all tasks and model classes by 3 % 3\% on average, indicating that EM is an useful method for test time scaling. We also note that EM can be applied to all tasks (math, coding and ugphysics) without assumptions such as output extraction in self-consistency or output scoring. Next, we note that EM competitively performs against self-consistency while outperforming it on tasks requiring significant reasoning capabilities, e.g. AIME, and UGPysics by 4 % 4\% and 3.6 % 3.6\% , respectively. Unlike self-consistency which requires sampling multiple trajectories ( N > 1 N>1 ), EM-INF requires only one trajectory leading to lower computational requirements. These observations collectively indicate that pretraining performance can be improved via EM, and making the model more deterministic at inference time via unsupervised EM in-order to prefer fewer chain of thought options can lead to better performance.

Computational efficiency: Figure 2 compares the compute requirements of EM-INF with inference scaling baselines. We observe that EM-INF can achieve better performance while using a third of time. This observation suggests that EM-INF is a practical inference scaling method in compute bound scenarios, for online adaption without updating model parameters [ 81 , 9 ] . EM-INF can also be combined with self-consistency and sequential refinement to scale up (Appendix Β§ C.1 ).

## 5 Limitations: Dependence on Base Model Capability and Confidence

Math AMC AIME Minerva Olymp . Llama-3.1-8B-Instruct 40.6 18.1 1.0 22.4 15.7 W/ RLOO 52.0 25.3 3.3 26.8 20.7 EM-FT (N=1) 42.2 21.7 3.3 18.8 16.4 EM-RL-token (N=4) 41.0 18.1 5.6 19.1 15.6 EM-RL-sequence (N=4) 41.6 18.1 7.8 18.4 16.6 IndVal Qwen2.5-7B-Instruct 65.9 EM-FT (N=1) 66.0 EM-RL-token (N=4) 66.0 EM-RL-sequence (N=4) 66.3

To contextualize the applicability of EM, we probe its performance by 1) training on tasks with different bias as compared to pretraining; and 2) finetuning models lacking target reasoning behaviors.

Table 4 (right) evaluates EM on individualistic value reasoning ( IndVal ) [ 35 ] . Individualistic value reasoning is a difficult alignment task that evaluates LLMs’ ability to reason about an individual’s value preferences in new situations. Given known statements about a certain individual, the task is to predict if the user would or would not associate with a new situation (measured using accuracy). We observe that Qwen-2.5-7B struggles with the task, likely because LLMs’ are biased to coarse values acquired from pretraining which may not generalize to a large pluralistic audience [ 97 , 35 ] . As expected, we note that EM is not effective in this task, likely because model confidences are incorrectly biased to training priors that do not benefit the target task, as also observed in [ 99 , 31 ] .

Furthermore, Table 4 (left) compares entropy minimization ( EM-FT , EM-RL ) with supervised RL (RLOO) using the same training data, setup and evaluation as Sec. 3.3 with Llama-3.1-8b-Instruct as the base model [ 24 ] . We observe that while supervised RL can improve model performance, improvements from entropy minimization is less effective as compared to Qwen-2.5 and can sometimes hurt performance on math reasoning. We tie this observation to [ 21 ] who show that Llama models may lack reasoning behaviors initially as compared to Qwen models, as also observed in [ 84 ] . The lack of certain reasoning behaviors for math in the initial policy possibly leads to unreliable model confidences and eventually inaccurate entropy rewards [ 12 ] . This observation suggests that success of entropy minimization hinges on the assumption that the pre-trained model’s initial reasoning behaviors are critical for the capacity of improvement. Overall, the success of EM-FT and EM-RL must be interpreted with care, as it can be less effective depending on the task and base model choice.

## 6 Conclusion

We show that entropy minimization through unsupervised finetuning (EM-FT), RL with standalone negative entropy as reward (EM-RL) and inference time scaling through logit optimization (EM-INF) can improve LLMs on complex math, physics, and coding tasks without using any labeled data. We observe that through EM-FT and EM-RL, Qwen-2.5-7B can achieve comparable or better performance than RL methods like GRPO and RLOO. Through test time logit optimization, EM-INF enables Qwen-2.5-32B to outperform proprietary models like GPT4o on complex scientific coding without training or updating model parameters. Our findings suggest that many pretrained LLMs already possess strong reasoning ability which can be enhanced using EM alone. While successful, EM is less beneficial on tasks and models with different inductive biases as pretraining. We call for the inclusion of EM-FT, EM-RL and EM-INF as a baseline in the evaluation of future post-pretraining and inference-time scaling algorithms, to better attribute the source of improvements in them.

## Acknowledgments and Disclosure of Funding

This research used the DeltaAI advanced computing and data resource, which is supported by the National Science Foundation (award OAC 2320345) and the State of Illinois. DeltaAI is a joint effort of the University of Illinois Urbana-Champaign and its National Center for Supercomputing Applications.

This work was also in part supported by research awards from Apple and the Allen Institute for AI.

Research was also supported in part by US DARPA INCAS Program No. HR0011-21-C0165 and BRIES Program No. HR0011-24-3-0325, National Science Foundation IIS-19-56151, the Molecule Maker Lab Institute: An AI Research Institutes program supported by NSF under Award No. 2019897, and the Institute for Geospatial Understanding through an Integrative Discovery Environment (I-GUIDE) by NSF under Award No. 2118329. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of DARPA or the U.S. Government.

## Appendix A Proof for proposition 1

(Informal) Both adaptive temperature scaling and logit optimization reduce the entropy of a model’s output distribution. However, in high-uncertainty settings (i.e., when the model’s predictive distribution has high entropy), logit optimization can change the order of non-top logits. In contrast, temperature scaling preserves the order of logits and merely sharpens or flattens the distribution proportionally. Both temperature scaling and logit optimization will keep the token with the top logit and increase its probability, making them less likely to change the model’s behavior when it is highly confident.

Part 1: βˆ€ 𝒛 ∈ ℝ | 𝒱 | \forall{\bm{z}}\in\mathbb{R}^{|\mathcal{V}|} let p ​ ( 𝒛 ) = softmax ​ ( 𝒛 ) p({\bm{z}})=\text{softmax}({\bm{z}}) . Let β„‹ ⁑ ( p ) \mathcal{H}(p) denote the entropy of p p ; β„‹ = βˆ’ βˆ‘ i = 1 | 𝒱 | p i log p i \mathcal{H}=-\sum_{i=1}^{|\mathcal{V}|}p_{i}\log p_{i} . βˆƒ 𝒛 \exists{\bm{z}} , Ξ· > 0 \eta>0 and a , b ∈ [ 1 , … , | V | ] a,b\in[1,\dots,|V|] , such that 𝒛 a < 𝒛 b {\bm{z}}_{a}<{\bm{z}}_{b} and π’š a > π’š b {\bm{y}}_{a}>{\bm{y}}_{b} where π’š i = z i + Ξ· ​ p i ​ ( log ⁑ p i + β„‹ ) {\bm{y}}_{i}=z_{i}+\eta p_{i}(\log p_{i}+\mathcal{H}) , i = 1 , … , | 𝒱 | i=1,\dots,|\mathcal{V}| .

Proof: Consider π’š a βˆ’ π’š b = 𝒛 a βˆ’ 𝒛 b + Ξ· ⁑ ( g a βˆ’ g b ) {\bm{y}}_{a}-{\bm{y}}_{b}={\bm{z}}_{a}-{\bm{z}}_{b}+\eta(g_{a}-g_{b}) where g i = p i ​ ( log ⁑ p i + β„‹ ) g_{i}=p_{i}(\log p_{i}+\mathcal{H}) . g g is the gradient of entropy β„‹ \mathcal{H} w.r.t. 𝒛 {\bm{z}} .

We now show that βˆƒ 𝒛 , a , b \exists{\bm{z}},a,b such that z a < z b z_{a}<z_{b} and g a > g b g_{a}>g_{b} .

By Mean-Value Theorem βˆƒ c ∈ ( p a , p b ) \exists c\in(p_{a},p_{b}) such that g a βˆ’ g b = ( p a βˆ’ p b ) ​ ( log ⁑ c + β„‹ + 1 ) g_{a}-g_{b}=(p_{a}-p_{b})(\log c+\mathcal{H}+1) . When p a βˆ’ p b < e βˆ’ ( β„‹ + 1 ) p_{a}-p_{b}<e^{-(\mathcal{H}+1)} we have c < p b < e βˆ’ ( β„‹ + 1 ) ⟹ log ⁑ c + β„‹ + 1 < 0 ⟹ g a βˆ’ g b > 0 c<p_{b}<e^{-(\mathcal{H}+1)}\implies\log c+\mathcal{H}+1<0\implies g_{a}-g_{b}>0 (6) Thus, we can set an arbitrary large Ξ· \eta such that, z a βˆ’ z b + Ξ· ⁑ ( g a βˆ’ g b ) > 0 z_{a}-z_{b}+\eta(g_{a}-g_{b})>0 (7)

Part 2: b b cannot be arg ⁑ max i ⁑ z i \arg\max_{i}z_{i} .

Proof: Let j = arg ⁑ max i ⁑ 𝒛 i j=\arg\max_{i}{\bm{z}}_{i} , i.e. the index of the maximum logit. Since p j = max i ⁑ p i p_{j}=\max_{i}p_{i} we can bound it using the inequality: β„‹ ( p ) = βˆ’ βˆ‘ i p i log p i β‰₯ βˆ’ log p j β‡’ p j β‰₯ e βˆ’ β„‹ ⁑ ( p ) β‰₯ e βˆ’ ( β„‹ ⁑ ( p ) + 1 ) \mathcal{H}(p)=-\sum_{i}p_{i}\log p_{i}\geq-\log p_{j}\quad\Rightarrow\quad p_{j}\geq e^{-\mathcal{H}(p)}\geq e^{-(\mathcal{H}(p)+1)} (8) We now analyze how g i g_{i} behaves as a function of p i p_{i} . Let g ⁑ ( p i ) = p i ​ ( log ⁑ p i + β„‹ ) g(p_{i})=p_{i}(\log p_{i}+\mathcal{H}) and g β€² ​ ( p i ) = log ⁑ p i + β„‹ + 1 g^{\prime}(p_{i})=\log p_{i}+\mathcal{H}+1 . Thus, g ⁑ ( p i ) g(p_{i}) is monotonically increasing whenever g β€² ​ ( p i ) > 0 g^{\prime}(p_{i})>0 , i.e. when, p i > e βˆ’ ( β„‹ + 1 ) p_{i}>e^{-(\mathcal{H}+1)} (9) So no lower-probability index i i can have g i > g j g_{i}>g_{j} , i.e., the entropy gradient cannot reduce π’š j {\bm{y}}_{j} relative to others. Therefore, b β‰  j b\neq j , leading to the following behavior 0 < g a ≀ g b p a ∈ ( e βˆ’ β„‹ , p b ] \displaystyle 0<g_{a}\leq g_{b}\quad p_{a}\in(e^{-\mathcal{H}},p_{b}] (10) g a < 0 < g b p a ∈ ( 0 , e βˆ’ β„‹ ) \displaystyle g_{a}<0<g_{b}\quad p_{a}\in(0,e^{-\mathcal{H}}) (11)

## Appendix B Adaptive Temperature Scaling

## Appendix C Additional Results

### C.1 Combining EM-INF with Self-consistency and Iterative Self-refinement

We combine EM-INF with self-consistency and iterative self-refinement in Figure 3(a) , 3(b) . Since, EM-INF only optimizes the logits, it can be plugged into both methods easily. We observe that EM-INF can get the same performance as self-consistency (N=4) while using 3 time less FLOPs on AMC with Qwen 2.5 7B. As expected, EM-INF reduces the diversity of the generation, which causes it’s benefits to diminish over iterations. Self-consistency benefits the most when several diverse generations can be produces which lead to the same answer, however, EM-INF reduces diversity through entropy minimization [ 44 , 7 ] leading to smaller benefits with larger compute. EM-INF is most suitable for inference time scaling on high uncertainty tasks under compute bound scenarios.

### C.2 Qualitative analysis of EM-INF on SciCode

We conduct a qualitative analysis of EM-INF on SciCode through a case study where Qwen2.5-7B-Instruct with EM-INF successfully generates the correct code, while the base model fails. The problem and corresponding model generations are illustrated in Figure 4 . The prompt requires generating a symmetric matrix with increasing diagonal values, where each element is modified by the product of a user-defined noise factor and a normally distributed random number, followed by symmetrization via averaging with the matrix’s transpose.

The base Qwen2.5-7B-Instruct model produces a partially correct implementation of the init_matrix function. It correctly initializes the increasing diagonal and applies the symmetrization formula ( A = (A + A.T) / 2 ). However, it fails to apply the noise factor to all elements of the matrix. Specifically, the noise modification is applied only to the off-diagonal elements, leaving the diagonal entries (e.g., i+1 ) unaltered by the stochastic process prior to symmetrization. This error occurs because scientific coding is a difficult task where both coding and domain knowledge is required, therefore making the model’s output distribution uncertain.

By contrast, the generation using EM-INF produces the correct implementation that adheres to all constraints specified in the prompt. EM-INF reduces the entropy of the output distribution, which leads to more deterministic generations. The generated code by EM-INF , as expected, reduces the diversity in generation, leading to more concise implementations and omitting additional comments. These results suggest that EM-INF is particularly beneficial for scientific coding tasks like SciCode, where correctness and determinism are critical.