The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning — part 1/5

Cleaned-text code points [0, 15264). Bibliography omitted here; source.md includes it.
# The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning

###### Abstract

Entropy minimization (EM) trains the model to concentrate even more probability mass on its most confident outputs. We show that this simple objective alone, without any labeled data, can substantially improve large language models’ (LLMs) performance on challenging math, physics, and coding tasks. We explore three approaches: (1) EM-FT minimizes token-level entropy similarly to instruction finetuning, but on unlabeled outputs drawn from the model; (2) EM-RL : reinforcement learning with negative entropy as the only reward to maximize; (3) EM-INF: inference-time logit adjustment to reduce entropy without any training data or parameter updates. On Qwen-7B, EM-RL , without any labeled data, achieves comparable or better performance than strong RL baselines such as GRPO [ 68 ] and RLOO [ 1 ] that are trained on 60K labeled examples. Furthermore, EM-INF enables Qwen-32B to match or exceed the performance of proprietary models like GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro on the challenging SciCode benchmark [ 78 ] , while being 3x more efficient than self-consistency and sequential refinement. Our findings reveal that many pretrained LLMs possess previously underappreciated reasoning capabilities that can be effectively elicited through entropy minimization alone, without any labeled data or even any parameter updates. 1 1 1 Code: https://github.com/shivamag125/EM_PT

## 1 Introduction

Entropy minimization (EM) trains the model to concentrate even more probability mass on its most confident outputs without any labeled data or external supervision [ 23 , 60 , 25 ] . It operates on a key assumption and a simple intuition: if a model is reasonably capable, it is more likely to be correct when it is confident [ 56 ] . Modern large language models (LLMs) are pretrained on massive data and often undergo a dedicated annealing phase to enhance downstream task performance [ 59 , 3 , 92 , 77 ] . It should be, therefore, fairly reasonable to treat them as capable models even before any supervised finetuning (SFT) or reinforcement learning (RL) [ 90 ] . This naturally invites the questions: Can entropy minimization alone improve their performance? And how far can they go without any labeled data? This work answers both: yes, and surprisingly far, at least for reasoning tasks.

We present a systematic exploration of post-pretraining adaptation and inference-time scaling of LLMs, using entropy minimization alone without any labeled data. We propose three novel methods, each aligned with an established post-pretraining stage. (1) Unsupervised finetuning by directly minimizing token-level entropy ( EM-FT ) mirrors SFT and minimizes a token level loss, on unlabeled outputs sampled from the model conditioning on the input prompts [ 46 ] . We find that EM-FT achieves surprisingly strong performance on math and coding tasks, and can even outperform labeled GRPO and RLOO on LeetCode [ 26 ] (coding) and Minerva [ 42 ] (math).

(2) Reinforcement learning with a negative entropy reward ( EM-RL ) uses a reward signal based solely on entropy: the negative sum of token-level entropy across a rollout, adjusted by a constant baseline. This is analogous to the REINFORCE algorithm [ 76 , 1 ] but with entropy as the only supervision without any labeled data. We find that without any labeled data EM-RL can achieve competitive performance to RLOO and GRPO on most math and coding tasks while outperforming it on LeetCode, Minerva and AMC (math) [ 43 ] .

(3) Inference-time scaling through entropy minimization (EM-INF) optimizes the logits during each decoding step to reduce the entropy of the LLM’s distribution without any parameter update. We find that EM-INF works best in complex tasks with high uncertainty (e.g. AIME math [ 43 ] , UGPhysics [ 88 ] and SciCode [ 78 ] ). We observe that Qwen 32B [ 77 ] can outperform frontier models like GPT-4o on Scicode [ 78 ] and is 3x more efficient than inference scaling through self-consistency and sequential refinement.

While the results are promising, they must be interpreted with care (§ 5 ). First, EM is most effective when model confidence correlates with correctness, as in the experiments above. It is less suited for tasks like aligning with human values [ 35 ] , where confidence alone is not a reliable proxy for quality. Through experiments on individualistic value reasoning [ 35 ] , we observe that Qwen-2.5 struggles on the task without finetuning on labeled data, and further EM leads to no improvements. Second, the effectiveness of EM hinges on the assumption that the pretrained model is already capable in the tasks of interest. We observe that the improvements from EM-FT and EM-RL on Llama-3.1-8B are less substantial on math reasoning tasks compared to those on Qwen-2.5. This may be due to Llama-3.1-8B is less capable on these reasoning tasks.

The success and limitations of EM highlight the importance of the capabilities of the pretrained models, which is sometimes underappreciated, at least for reasoning tasks. We call for the inclusion of EM as a baseline in the evaluation of future post-pretraining and inference-time scaling algorithms, to better separate the contribution of algorithmic innovations from the intrinsic capabilities of the model itself. Besides, as we show, an approach that performs surprisingly well on certain models and tasks can fail in trivial ways on others. Broad evaluation across diverse pretrained models and tasks can help ensure that the conclusions generalize and are not artifacts of a specific model or dataset.

## 2 Background and Related Work

### 2.1 Related Work

Entropy-regularized RL augments the expected reward objective with a regularization term aiming to increase entropy [ 98 ] , in order to promote exploration by adding uncertainty into the actions [ 64 , 33 , 27 , 18 , 41 ] . Entropy-regularized RL usually learns a policy from scratch in grid world situations (e.g. robot simulations) where adding perturbations can lead to exploring various paths to get to the goal. In contrast, we investigate if reducing the uncertainty of the policy can improve a pre-trained language models capabilities, with the hypothesis that they already acquire some capabilities for the tasks of interest during pretraining. Instead of learning from scratch, we finetune LLMs to reinforce confident outputs. Therefore, we draw inspiration from entropy-regularization in RL to design reward function ( EM-RL ) that minimize the entropy of the model encouraging it to commit to a fewer more confident paths, instead of prompting exploration like in previous works.

Entropy minimization (EM) has been extensively used as regularization in semi-supervised learning [ 23 , 40 , 8 , 51 ] , domain adaptation [ 60 , 62 , 80 ] , and few-shot learning [ 25 , 39 ] . [ 38 , 20 ] apply EM for reinforcement learning with semi-supervised scenarios, where entropy is used to estimate the rewards on unlabeled data. [ 81 , 94 , 56 ] use EM for unsupervised test-time adaptation, where pre-trained classifiers are adapted to new data at test time without using any labels. Despite its well-studied history [ 53 , 52 , 48 , 15 , 61 , 71 ] , EM has rarely been explored as a standalone training objective for LLMs. Our key hypothesis is that EM can reinforce LLMs’ strong capabilities acquired during pretraining. Instead of applying entropy minimization for test time-domain adaptation or as a regularizer, we study if it can be used for LLM finetuning and inference time scaling.

Unsupervised and few-shot RL for LLMs has raised the possibility of self-improvement [ 85 , 5 , 54 ] . Here a model learns via self-training in which it critiques its own generations without external feedback [ 22 , 47 , 57 , 93 ] or refines generated text [ 11 , 96 , 70 ] . Building on self-improvement, self-verification [ 45 ] has been applied for RL tuning [ 95 , 84 , 99 ] , concurrently. [ 32 ] uses the most frequent answer as reward (majority voting) for finetuning, since optimizing the likelihood of most consistent answer can reinforce pretraining priors. [ 99 ] applies this idea to unsupervised RL tuning at test time by optimizing on test dataset as a whole. We work on the same intuition of reinforcing model confidence, however, we study post-training where EM-FT and EM-RL are used for finetuning LLMs on large general math and coding data. Further, consistent answers and output extraction is not always possible in complex tasks like scientific coding [ 2 , 10 , 82 , 4 , 73 ] . On the other hand, EM-FT and EM-RL can be applied on tasks like code generation where output extraction is non-trivial. Unlike [ 99 ] which updates the model parameters on each test set individually, inference scaling via EM-INF optimizes the models’ entropy without updating parameters and does not need task-specific optimization. EM-INF can be deployed and served like regular text decoding [ 30 , 58 , 19 ] requiring neither assumption on task nor model optimization. [ 95 ] proposes to estimate rewards by computing the frequency of extracted answers for a prompt. Here, the model is incentivized to prefer certain answers, which makes the answer distribution more concentrated. While it reduces the entropy, it is not directly enforced and emerges from learning to repeat most frequent solutions. EM-FT and EM-RL target uncertainty in models’ token level distribution, whereas the frequency-based approaches target alignment with desired outputs where entropy reduction is not an inherent outcome and task-dependent. [ 84 ] investigates RL from few-shot output verified labeled data and observe that direct entropy maximization as a standalone objective can be used for finetuning to a small extent. They find that carefully selecting a few output-verifiable problems along with entropy maximization as a regularizer can lead to improvements. We find that only negative entropy rewards (entropy minimization) are sufficient for RL with similar effects without requiring any data selection. Standalone entropy maximization does not converge and eventually collapses [ 84 ] , but EM-RL and EM-FT via EM converge and can be used to train on large datasets without output verification.

### 2.2 Preliminaries

Problem setup: Let π θ \pi_{\theta} denote an autoregressive LLM policy: π θ  ( 𝒚 | 𝒙 ) = ∏ i π θ  ( y i | 𝒙 , 𝒚 < i ) \pi_{\theta}({\bm{y}}|{\bm{x}})=\prod_{i}\pi_{\theta}(y_{i}|{\bm{x}},{\bm{y}}_{<i}) , where 𝒙 {\bm{x}} denotes the input prompt and 𝒚 = y 1  …  y | 𝒚 | {\bm{y}}=y_{1}\dots y_{|{\bm{y}}|} the output trajectory. For clarity, we will suppress the conditioning on 𝒙 {\bm{x}} whenever it is clear from the context. Let r ⁡ ( 𝒚 ) r({\bm{y}}) denote a reward function, which provides a score for an output sequence. RL learns a policy that produces output sequence to maximize the expected reward 𝔼 y ∼ π θ  [ r  ( 𝒚 ) ] \mathbb{E}_{y\sim\pi_{\theta}}[r({\bm{y}})] .

Entropy minimization minimizes the Shannon entropy ℋ ⁡ ( π θ ) = − 𝔼 𝒚 ∼ π θ  [ log ⁡ π θ  ( 𝒚 ) ] \mathcal{H}(\pi_{\theta})=-\mathbb{E}_{{\bm{y}}\sim\pi_{\theta}}[\log\pi_{\theta}({\bm{y}})] [ 67 ] of the policy [ 23 ] . We consider two methods for empirically estimating the entropy.

Trajectory-level entropy estimator [ 79 , 17 ] is computed based on the trajectory distribution of the output sequences produced by the policy. In practice, it samples full sequences and computes the total log-probability. For an autoregressive policy, ℋ ^ traj ( π θ ) = − 1 N ∑ i = 1 N log π θ ( 𝒚 i ) = − 1 N ∑ i = 1 N ∑ t = 1 | 𝒚 i | log π θ ( y t i | 𝒚 < t i ) \displaystyle\hat{\mathcal{H}}_{\text{traj}}(\pi_{\theta})=-\frac{1}{N}\sum_{i=1}^{N}\log\pi_{\theta}\left({\bm{y}}^{i}\right)=-\frac{1}{N}\sum_{i=1}^{N}\sum_{t=1}^{|{\bm{y}}^{i}|}\log\pi_{\theta}(y_{t}^{i}|{\bm{y}}_{<t}^{i}) (1) { 𝒚 i ∼ π θ } i = 1 N \{{\bm{y}}^{i}\sim\pi_{\theta}\}_{i=1}^{N} are N N full trajectories sampled from the policy.

Token (action) level entropy estimator [ 27 , 18 ] computes the entropy of the policy based on the token (action) distribution at each step. In practice, it sums per-token entropy across time steps. At generation step t t , the model produces a probability distribution π θ ( ⋅ | 𝒚 < t ) \pi_{\theta}(\cdot|{\bm{y}}_{<t}) over a vocabulary 𝒱 \mathcal{V} . The entropy of the policy using token-level estimation is then ℋ ^ tok ( π θ ) = 1 N ∑ i = 1 N ∑ t = 1 | 𝒚 i | ℋ ( π θ ( ⋅ ∣ 𝒚 < t i ) ) \displaystyle\hat{\mathcal{H}}_{\text{tok}}(\pi_{\theta})=\frac{1}{N}\sum_{i=1}^{N}\sum_{t=1}^{|{\bm{y}}^{i}|}\mathcal{H}\big(\pi_{\theta}(\cdot\mid{\bm{y}}_{<t}^{i})\big) (2) where ℋ ( π θ ( ⋅ ∣ 𝒚 < t ) ) = − ∑ j ∈ 𝒱 π θ ( j ∣ 𝒚 < t ) log π θ ( j ∣ 𝒚 < t ) \mathcal{H}(\pi_{\theta}(\cdot\mid{\bm{y}}_{<t}))=-\sum_{j\in\mathcal{V}}\pi_{\theta}(j\mid{\bm{y}}_{<t})\log\pi_{\theta}(j\mid{\bm{y}}_{<t}) . While both trajectory-level and token-level estimators are used to compute entropy, they lead to different behaviors when used for training. Minimizing trajectory entropy leads to policies with lower entropy over trajectories whereas minimum token-level entropy leads to policies with low entropy at each generation step. Furthermore, the token-level estimate is commonly used in maximum entropy-regularized RL frameworks, such as soft actor-critic due to its low variance [ 14 , 27 , 64 , 41 ] .

In the following subsections, we introduce three post-training methods based on entropy minimization as a standalone objective. (1) EM-FT (§ 3.1 ): directly minimizes the token-level entropy (Eqn. 2 ) similar to supervused finetuning, but on unlabeled outputs drawn from the model; (2) EM-RL (§ 3.2 ): reinforcement learning with negative trajectory-level entropy (Eq. 1 ) or neagtive token level entropy (Eqn. 2 ) as the only reward to maximize; (3) EM-INF (§ 4 ): inference-time logit adjustment to reduce entropy using the token level estimate (Eqn. 2 ) without any training data or parameter updates.

## 3 Entropy Minimization for Post-training

### 3.1 EM-FT : Direct Entropy Minimization through Token-Level Finetuning

Supervised finetuning methods such as rejection sampling finetuning [ 16 , 91 , 46 ] sample data from the policy and use a reward function to select high-quality data. Good performance on these tasks has been attributed to post-training labels (e.g. output supervision) and pretraining task mixtures [ 65 , 89 ] . Building on entropy minimization and on-policy finetuning we test if directly minimizing the entropy of the sampled trajectories can enable learning without labels. Given a batch of N N sampled trajectories { 𝒚 ∼ π θ } i = 1 N \{{\bm{y}}\sim\pi_{\theta}\}_{i=1}^{N} , we directly minimize the the empirical total token-level entropy ℋ ^ tok \hat{\mathcal{H}}_{\text{tok}} (Eqn. 2 ). We present this objective and its corresponding gradient in Table 1 .

### 3.2 EM-RL : Entropy Minimization with Negative Entropy Rewards in RL

We now minimize the entropy of the policy by using negative entropy as the only reward in RL ( EM-RL ). We design a reward function to assign outcome rewards to trajectory 𝒚 {\bm{y}} based on: 1) negative trajectory level entropy ( EM-RL -sequence); 2) negative token level entropy ( EM-RL -token).