The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning β part 2/5
Cleaned-text code points [15264, 31174). Bibliography omitted here; source.md includes it.
EM-RL -sequence minimizes the entropy based on the trajectory level estimator of entropy (Eqn. 1 ). This estimate favors trajectories that receive high probabilities but does not care about if the reasoning paths are concentrated or not to a few options. Minimizing this entropy estimate is useful in situations where multiple but limited chain of thoughts are preferred, for example, shorter math tasks where some plausible solutions can be explored but not too many. We use the joint probability of the sequence as a reward r traj r_{\text{traj}} to minimize the trajectory level entropy estimate in expectation, πΌ π βΌ Ο ΞΈ ] [ r traj ( π ) ] = β β ^ traj ( Ο ΞΈ ) \mathbb{E}_{{\bm{y}}\sim\pi_{\theta}]}[r_{\text{traj}}({\bm{y}})]=-\hat{\mathcal{H}}_{\text{traj}}(\pi_{\theta}) . The reward r traj r_{\text{traj}} is, r traj β ( π ) = β t = 1 | π | log β‘ Ο ΞΈ β ( y t | y < t ) = log β‘ Ο ΞΈ β ( π ) r_{\text{traj}}({\bm{y}})=\sum_{t=1}^{|{\bm{y}}|}\log\pi_{\theta}(y_{t}|y_{<t})=\log\pi_{\theta}({\bm{y}}) (3)
EM-RL -token minimizes the entropy based on the token level entropy estimate. This estimate prefers trajectories that are more deterministic and confident at each generation step, focusing the probability mass on fewer possible reasoning paths (chain of thoughts). For example, problems requiring complex long chain of thoughts where being more deterministic at every step prevents the model getting lost in the sauce. Based on the token level entropy estimate (Eqn 2 ), we define the reward r tok r_{\text{tok}} to minimize the token level entropy estimate in expectation, πΌ π βΌ Ο ΞΈ β [ r tok β ( π ) ] = β β ^ tok β ( Ο ΞΈ ) \mathbb{E}_{{\bm{y}}\sim\pi_{\theta}}[r_{\text{tok}}({\bm{y}})]\!=\!-\hat{\mathcal{H}}_{\text{tok}}(\pi_{\theta}) , r tok ( π ) = β β t = 1 | π | β ( Ο ΞΈ ( β
β£ y < t ) ) r_{\text{tok}}({\bm{y}})=-\sum_{t=1}^{|{\bm{y}}|}\mathcal{H}\big(\pi_{\theta}\left(\cdot\mid y_{<t}\right)\big) (4) While both the objectives EM-FT , EM-RL aim to minimize the entropy of the policy, they are optimized differently. EM-FT minimizes the entropy by directly differentiating it, whereas EM-RL uses policy gradients [ 64 , 1 ] . Table 1 summarizes the objectives and the corresponding gradients. In practice, we subtract the RLOO baseline [ 1 ] in EM-RL , which does not change the expectation of the gradients but helps with optimization [ 75 , 49 , 87 ] . In addition, we use a KL-regularizer weighted by Ξ² \beta between the policy Ο ΞΈ \pi_{\theta} and the initial base model Ο ref \pi_{\text{ref}} to avoid large deviations from the base model and aid optimization, i.e., β β ( Ο ΞΈ ) β Ξ² KL [ Ο ΞΈ | | Ο ref ] = β β ( Ο ΞΈ ) β Ξ² ( β β ( Ο ΞΈ ) β β ( Ο ΞΈ , Ο ref ) ) -\mathcal{H}(\pi_{\theta})-\beta\text{KL}[\pi_{\theta}||\pi_{\text{ref}}]=-\mathcal{H}(\pi_{\theta})-\beta(-\mathcal{H}(\pi_{\theta})-\mathcal{H}(\pi_{\theta},\pi_{\text{ref}})) [ 64 , 74 ] . The KL regularization can increase the entropy when the base model is too uncertain (higher entropy), however when Ξ² < 1 \beta<1 (we use 0.001 0.001 ) the dominant gradient is from the reward, in our case entropy minimization.
Objectives Gradients wrt. ΞΈ \theta Finetuning EM-FT Β§ 3.1 Min. β ^ tok \hat{\mathcal{H}}_{\text{tok}} 1 N β i = 1 N β t = 1 | y i | β ΞΈ β ( Ο ΞΈ ( β
β£ π < t i ) ) \frac{1}{N}\sum_{i=1}^{N}\sum_{t=1}^{|y^{i}|}\nabla_{\theta}\mathcal{H}\left(\pi_{\theta}(\cdot\mid{\bm{y}}_{<t}^{i})\right) RL EM-RL-sequence Β§ 3.2 Max. β β ^ traj -\hat{\mathcal{H}}_{\text{traj}} 1 N β i = 1 N [ β t = 1 | y i | log Ο ΞΈ ( y t i β£ π < t i ) β ΞΈ log Ο ΞΈ ( π i ) ) ] \frac{1}{N}\sum_{i=1}^{N}\Big[\sum_{t=1}^{|y^{i}|}\log\pi_{\theta}(y_{t}^{i}\mid{\bm{y}}_{<t}^{i})\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{i}))\Big] RL EM-RL-token Β§ 3.2 Max. β β ^ tok -\hat{\mathcal{H}}_{\text{tok}} 1 N β i = 1 N [ β β t = 1 | y i | β ( Ο ΞΈ ( β
β£ π < t i ) β ΞΈ log Ο ΞΈ ( π i ) ) ] \frac{1}{N}\sum_{i=1}^{N}\Big[-\sum_{t=1}^{|y^{i}|}\mathcal{H}(\pi_{\theta}(\cdot\mid{\bm{y}}_{<t}^{i})\nabla_{\theta}\log\pi_{\theta}({\bm{y}}^{i}))\Big] Inf-Scaling EM-INF Β§ 4 Min. β ^ tok \hat{\mathcal{H}}_{\text{tok}} N/A
Connecting to entropy regularized RL: Entropy-regularized RL by previous works [ 27 , 18 ] uses a an entropy maximization term (as a regularizer) along with the reward signal (policy loss) to encourage stochasticity in action selection and promote exploration. [ 84 ] build on this idea and use few labeled examples for the policy loss along with entropy maximization. They find that using entropy maximization only can boost performance initially, however, the optimization with only entropy maximization does not converge and eventually collapses the policy. In contrast, EM-RL aims to reinforce pretraining biases by learning a more deterministic policy via entropy minimization without requiring any labeled examples. We find that EM-RL can converge on large datasets and improve model performance on math and code generation.
### 3.3 Training Setup
We use the following setup for all experiments, and provide additional details in Appendix D .
Evaluation: For our training experiments, we evaluate on math and coding tasks using [ 13 ] - Math-500 [ 29 ] , AMC [ 43 ] , AIME 2024 [ 43 ] , Minerva math (Minerva) [ 42 ] , Olympiad Bench ( Olymp. ) [ 28 ] , LeetCode ( LeetC ) [ 26 ] , and LiveCodeBench-v2 ( LiveC ) [ 34 ] . For inference time scaling, we additionally evaluate on Scicode [ 78 ] and UGPhysics [ 88 ] . Following [ 63 , 37 ] , we estimate the training FLOPs as 6 β P β D 6PD and inference FLOPs as 2 β P β D 2PD where P P is the number of parameters in the model and D D is the number of tokens (Appendix Β§ D.4 )
Training data, models, and methods: We construct the training data for math and coding by randomly sampling 35K prompts from Numina math [ 43 ] and 25K prompts from Eurus-2 coding split [ 13 ] , respectively. We use Qwen2.5-Math-7B [ 89 ] and Eurus-2-7B-SFT [ 13 ] model for training on math and coding, respectively. We use Verl [ 69 ] to train. For all RL methods, we use early stopping based on the validation set. We set the number of rollouts to N = 4 N=4 , batch size to 512 512 , learning rate to 1 β e β 6 1e^{-6} . We use a KL regularizer with small coefficient Ξ² = 0.001 \beta=0.001 which does not effect entropy minimization. We use RLOO and GRPO with output verification only without reward models. We use the same prompts for EM-FT but without the labels. Instead it samples solutions to estimate the entropy. All experiments are done using 4xGH200 Nvidia GPUs.
Baseline: We consider an additional rewarding scheme based on the frequency of the final answerβ Self consistency (SC-RL) . It assigns rewards to each trajectory π i {\bm{y}}_{i} based on majority voting [ 85 , 32 ] . An answer a i a_{i} is extracted for the trajectory π i {\bm{y}}_{i} and the frequency of the answer is used estimate the reward, r β‘ ( π i ) = β j = 1 N π β‘ ( a i = a j ) N r({\bm{y}}_{i})=\sum_{j=1}^{N}\frac{\mathbb{I}(a_{i}=a_{j})}{N} , where π \mathbb{I} is the indicator function. Self-consistency assigns hard rewards to trajectories where answer extraction is possible (e.g. math), however is inapplicable on tasks like code generation. Such rewards are useful on tasks where several consistent answers can be generated (e.g. MATH-500). However, challenging prompts (e.g. scientific coding) often admit multiple plausible solutions with no answer extraction possible, consequently making majority voting inapplicable. [ 99 ] apply this idea for training on the test set, however, we use it on the training set.
Math Coding Math AMC AIME Minerva Olymp. Avg. LeetC LiveC Avg. FLOPs Qwen2.5-7b 43.8 31.3 15.6 14.7 19.0 24.9 26.1 18.4 22.3 β Trained using 60K labeled prompts w/ SFT N=1 48.2 30.2 10.0 17.6 22.4 25.7 18.3 18.3 18.3 0 1.0 w/ RLOO N=4 73.0 57.8 23.3 31.2 34.2 43.9 28.3 26.7 27.5 13.1 w/ GRPO N=4 71.8 56.6 21.1 25.0 35.9 42.1 25.0 25.8 25.4 13.1 Our unsupervised methods trained on 60K unlabeled prompts EM-FT N=1 67.2 51.8 14.4 33.1 34.4 40.2 28.3 17.2 22.8 0 1.0 SC-RL N=4 73.2 51.8 15.6 26.1 36.7 40.7 β β β 13.1 EM-RL-sequence N=4 67.2 53.0 21.1 30.9 35.6 41.6 31.1 21.7 26.4 13.1 EM-RL-token N=4 70.8 57.8 18.9 30.9 35.9 42.9 29.5 24.5 27.0 13.1
### 3.4 Results: Entropy Minimization for Post-Training
Table 2 compares the performance of using entropy minimization as a standalone post-training objective to supervised finetuning and output-verified RL.
Can unsupervised finetuning by directly minimizing token-level entropy improve LLMs? We first observe that directly minimizing the entropy (EM-FT) can improve improve performance over the base model by 8 % 8\% on average on math and coding without using any labels. This observation indicates that pretraining priors can be further capitalized to squeeze more performance and entropy can independently contribute to performance gains. We also note that with only N = 1 N=1 trajectory sampled from the model, EM-FT can achieve competitive performance to GRPO and RLOO (outperforms on Minerva and LeetCode by 5 % 5\% on average) which require output verification over multiple sampled ( N = 4 N=4 ) trajectories. EM-FT better contextualizes the improvements from supervised methods and better uses pretraining capacity, while maintaining a smaller computational footprint. We also observe no improvements on LiveCode likely because LiveCode is regularly updated with new problems that might be different from the pretraining distribution [ 34 ] . This observation indicates that entropy minimization can potentially boost performance on unlabeled scenarios if itβs not significantly different from the pretraining task mixture [ 81 , 73 , 31 ] .
Can we improve learning from entropy minimization through RL? In Table 2 , we observe that unsupervised RL methods ( SC-RL,EM-RL ) improves base model performance by 11 % 11\% on average and maintains competitive performance as compared to output-verified GRPO and RLOO (outperforms on AMC, Minerva and LeetCode by 4.5 % 4.5\% on average) under similar compute constraints. This observation suggests that model confidences in the form of token and sequence level entropy estimation can be used as pseudo-rewards to learn from unlabeled scenarios and better use pretraining priors. This observation also ties up with [ 55 , 20 , 31 ] who show that model confidences can be a proxy for reward. Among the various rewarding schemes, we observe that token level and sequence level entropy estimation can be flexibly applied to a variety of tasks without assumptions on the problem structure (e.g. answer extraction) unlike self-consistency which is not applicable for code generation. On average, we observe that EM-RL has better performance via entropy rewards, indicating that fewer deterministic CoTs is more useful on certain reasoning tasks (LiveCode, AMC). We also notice that EM-RL improve model performance on LiveCode by 6 % 6\% where direct entropy minimization ( EM-FT ) did not work, indicating that careful reward design to learn from unlabeled experience can better enhance learning and pretraining capabilities. Through EM-FT and EM-RL , we show that EM can elicit pretraining priors necessary for reasoning without any labeled data. While supervised training is expected to scale better, we put forward EM-FT and EM-RL as important baselines to better contextualize improvements from post-training methods.
Math Coding Science Math AMC AIME Minerva Olymp. Avg. LeetCode UGPhysics Qwen2.5-Math-7B-Instruct 79.2 48.2 0 8.9 43.4 41.3 44.2 0 0.6 18.6 Greedy Decoding ( t = 0 t=0 ) 79.0 53.0 11.1 42.6 39.9 45.1 0 1.1 18.8 Iterative-refinement ( N = 3 N=3 ) 79.2 48.2 0 8.9 43.4 41.3 44.2 0 1.7 16.4 Self-consistency ( N = 4 N=4 ) 78.8 53.0 11.1 43.4 40.7 45.4 β 19.6 Adaptive Temp 80.8 55.4 10.0 41.5 38.7 45.3 0 2.2 23.7 EM-INF 80.2 54.2 14.4 42.3 40.8 46.4 0 0.6 23.1 Qwen2.5-7B-Instruct 74.0 41.0 0 8.9 41.2 37.5 40.5 47.2 23.4 Greedy Decoding ( t = 0 t=0 ) 74.2 44.6 0 7.8 40.4 37.2 40.8 46.1 23.7 Iterative-refinement ( N = 3 N=3 ) 73.8 37.3 10.0 41.5 37.8 40.1 50.0 21.5 Self-consistency ( N = 4 N=4 ) 73.4 43.4 10.0 33.5 41.3 40.3 β 23.4 Adaptive Temp 74.4 50.6 10.0 41.9 39.7 43.3 49.4 25.8 EM-INF 73.8 45.8 11.1 41.2 38.2 42.0 51.7 26.9 Llama-3.1-8B-Instruct 40.6 18.1 0 1.1 22.4 15.7 19.6 10.6 17.5 Greedy Decoding ( t = 0 t=0 ) 40.6 16.9 0 3.3 21.0 16.0 19.6 12.8 16.1 Iterative-refinement ( N = 3 N=3 ) 41.0 19.3 0 1.1 22.4 15.7 19.9 11.7 18.3 Self-consistency ( N = 4 N=4 ) 41.2 20.5 0 4.4 20.2 19.4 21.1 β 20.2 Adaptive Temp 43.6 25.3 0 5.5 24.3 16.6 23.1 17.2 19.9 EM-INF 43.0 22.9 0 3.3 22.8 16.4 21.7 14.4 20.4
## 4 EM-INF: Inference-time Logit Optimization
Inference-time scaling methods suggest that enabling LLMs to improve their output generations at test time by scaling compute is critical for improving performance on a complex tasks [ 86 , 72 , 66 , 50 , 6 ] . Practical LLMs need to adapt to tasks online, i.e., we should be able to adapt to user-queries while the requests are being received. As a result, online LLM adaptation should be unsupervised. Self-consistency (parallel generations) [ 83 ] and iterative-refinement (sequential generations) are widely used for reasoning when external verification is not available [ 47 ] . Self-consistency samples multiple answers from the model and then selects the solution with the largest frequency. Whereas iterative-refinement feeds the previously generated solution into the model as in-context examples inorder to refine the solution. However, online adaptation should not be tailored to certain problem typesβself-consistency only works when output extraction is possible and iterative refinement is bottle-necked by context length. To tackle these challenges we propose EM-INF that uses entropy minimization at inference time for logit adjustment. EM-INF treats the modelβs output logits as if they were parameters, and uses gradient descent to update them to minimize the entropy of the distribution they induce; no gradient wrt. or the model parameters or parameter update is needed.
Let π t β β π± {\bm{z}}_{t}\in\mathbb{R}^{\mathcal{V}} be a the logit vector produced by the model at generation step t t from its last layer. We freeze the model parameters and optimize the logits π t {\bm{z}}_{t} to minimize the entropy of the output distribution it induces via gradient descent. We treat the logits as free parameters and never backpropagate through the model or update its parameters. To prevent over-optimization, which would effectively lead to greedy decoding, we use a minimum entropy threshold Ξ΄ \delta (we empirically find 0.1 < Ξ΄ < 0.5 0.1<\delta<0.5 works the best). Our inference time objective for each generation steps is defined as, β EM-INF = max ( β β j β π± Ο ( π t ) j log Ο ( π t ) j , Ξ΄ ) , \mathcal{L}_{\text{EM-INF}}=\max\Big(-\sum_{j\in\mathcal{V}}\sigma({\bm{z}}_{t})_{j}\log\sigma({\bm{z}}_{t})_{j},\delta\Big), (5) where Ο \sigma denotes the softmax function. After logit optimization, we use sampling based decoding to select the next token, since this optimization does not change the top logit and greedy decoding will not change the output post-optimization (Prop. 1 ) [ 31 ] . The above optimization operates over π t {\bm{z}}_{t} and is equivalent to optimizing a one-layer neural network with | π± | |\mathcal{V}| parameters with a softmax activation. We empirically find that 5 to 15 gradient steps are enough. The vocabulary size ( βΌ 150 \sim 150 K in Qwen2.5 [ 77 ] ) is much smaller than number of parameters ( 7 7 B) in the model, making the optimization overhead negligible compared to a forward pass of the full model. We require only πͺ β‘ ( n ) \mathcal{O}(n) ( n n is sequence length) forward passes from the language model, equivalent to regular decoding. Whereas, other inference time scaling methods like self-consistency and sequential refinement require πͺ β‘ ( N β n ) \mathcal{O}(Nn) forward passes where N N is the number of trajectories.