Evolutionary Strategies lead to Catastrophic Forgetting in LLMs — part 2/2

Cleaned-text code points [15432, 28062). Bibliography omitted here; source.md includes it.

Therefore, GRPO exhibits structured and comparatively sparse updates, aligning with the hypothesis that gradient-based optimization concentrates changes in task-relevant subspaces and mitigates interference with prior capabilities. When combined with KL regularization, these mechanisms provide a natural safeguard against large-scale parameter drift and, consequently, catastrophic forgetting. In contrast, we see that updates using ES have orders of magnitude larger norms and are much less sparse compared to GRPO. The lack of sparsity and large update norms in ES drifts the fine-tuned model further away from the base model, potentially leading to the catastrophic forgetting behavior observed in previous sections.

## 4 Conclusion

We perform an empirical analysis of Evolutionary Strategies for fine-tuning LLMs based on recent work Qiu et al. (2025) and show that it performs competetively with GRPO. Although, a critical roadblock still persists: we observe that ES exhibits significant catastrophic forgetting and progressively deteriorates performance on prior skills of the model. We show that this happens because ES updates have large norms and low sparsity levels (more dense), resulting in parameter drifts that are 1000x higher in magnitude than drifts observed with GRPO for the same number of update steps. These results imply that although recent progress in ES has bridged performance gap with state-of-the-art learning algorithms like GRPO, its intense model degradation still remains a challenge before its widespread adoption.

## Limitations

ES has an inherent randomness associated with the updates due to the nature of the algorithm. As a result, models trained with ES exhibit high variance in stochastic perturbations. Although we observed consistent qualitative trends across the settings we tested, where we worked with a population size of 30 as suggested in prior work Qiu et al. (2025) , increased population size will decrease the variance and increase statistical stability of our performance numbers.

Additionally, we evaluate catastrophic forgetting by tracking performance on one task during continued fine-tuning on Countdown, which measures retention of a broad prior capability. Doing so does not fully capture multi-facetted loss of performance that may be happening in the model; however, is enough to give strong evidence of the occurence of the phenomenon.

## Appendix A Appendix

### A.1 Algorithmic Overview and Analogy Between ES and GRPO

#### A.1.1 ES Algorithm Overview

Qiu et al. (2025) implement a version of evolutionary strategies that features these techniques: weight adjustment in-place with noise generation from stored random seeds, ranked weight updates, and learning rate ingestion.

Each update step can be understood through the following equations.

Each population member at step t t has a unique seed. With noise per iteration ϵ n , l ∼ 𝒩 ⁡ ( 0 , I ) \epsilon_{n,l}\sim\mathcal{N}(0,I) , model parameters for timestep t t θ t \theta_{t} , layer parameters for step t t θ t , l \theta_{t,l} , reward function R R , reward score for the n n th member R n R_{n} , z-score for n n th member Z n Z_{n} , and noise coefficient σ \sigma , and learning rate α \alpha .

Reset random seed generator. Sample noise ϵ n , l ∼ 𝒩 ⁡ ( 0 , I ) \epsilon_{n,l}\sim\mathcal{N}(0,I) . For all layers, perturb in-place: θ t − 1 , l ← θ t − 1 , l + σ ⋅ ϵ n , l . \theta_{t-1,l}\leftarrow\theta_{t-1,l}+\sigma\cdot\epsilon_{n,l}.

Reward for perturbed model is calculated: R n = R ⁡ ( θ t − 1 ) . R_{n}=R(\theta_{t-1}).

Reset random seed generator. Sample noise ϵ n , l ∼ 𝒩 ⁡ ( 0 , I ) \epsilon_{n,l}\sim\mathcal{N}(0,I) . For all layers, restore in-place: θ t − 1 , l ← θ t − 1 , l − σ ⋅ ϵ n , l . \theta_{t-1,l}\leftarrow\theta_{t-1,l}-\sigma\cdot\epsilon_{n,l}.

Z-score is calculated per population member: Z n = R n − R mean R std , Z_{n}=\frac{R_{n}-R_{\text{mean}}}{R_{\text{std}}},

Reset random seed generator. Sample noise ϵ n , l ∼ 𝒩 ⁡ ( 0 , I ) \epsilon_{n,l}\sim\mathcal{N}(0,I) . For all layers, update with noise weighted by z-score and learning rate in-place: θ t , l ← θ t − 1 , l + α ⋅ 1 N ​ Z n ​ ϵ n , l . \theta_{t,l}\leftarrow\theta_{t-1,l}+\alpha\cdot\frac{1}{N}Z_{n}\epsilon_{n,l}.

where R mean R_{\text{mean}} and R std R_{\text{std}} are the mean and standard deviation of R 1 , R 2 , … , R N R_{1},R_{2},\dots,R_{N} .

#### A.1.2 ES Algorithm Overview

Shao et al. (2024) implement Group Relative Policy Optimization (GRPO), which eliminates the critic model by estimating advantages from group statistics.

For each prompt q q , sample a group of G G outputs { o 1 , o 2 , … , o G } \{o_{1},o_{2},\dots,o_{G}\} from the current policy π θ o ​ l ​ d \pi_{\theta_{old}} .

Compute rewards { r 1 , r 2 , … , r G } \{r_{1},r_{2},\dots,r_{G}\} for each output and calculate relative advantages via z-score normalization: A i = r i − mean ​ ( { r 1 , … , r G } ) std ​ ( { r 1 , … , r G } ) . A_{i}=\frac{r_{i}-\text{mean}(\{r_{1},\dots,r_{G}\})}{\text{std}(\{r_{1},\dots,r_{G}\})}.

The policy π θ \pi_{\theta} is updated by maximizing the GRPO objective:

ρ i ​ ( θ ) = π θ ​ ( o i ∣ q ) π θ old ​ ( o i ∣ q ) \rho_{i}(\theta)=\frac{\pi_{\theta}(o_{i}\mid q)}{\pi_{\theta_{\text{old}}}(o_{i}\mid q)}

𝒥 GRPO ​ ( θ ) \displaystyle\mathcal{J}_{\text{GRPO}}(\theta) = 1 G ∑ i = 1 G min ( ρ i ( θ ) A i , \displaystyle=\frac{1}{G}\sum_{i=1}^{G}\min\Big(\rho_{i}(\theta)A_{i},\; OPEN clip ​ ( ρ i ​ ( θ ) , 1 − ϵ , 1 + ϵ ) ​ A i ) \displaystyle\text{clip}(\rho_{i}(\theta),1-\epsilon,1+\epsilon)A_{i}\Big) − β 𝔻 KL ( π θ ∥ π ref ) . \displaystyle-\beta\,\mathbb{D}_{\mathrm{KL}}(\pi_{\theta}\,\|\,\pi_{\text{ref}}).

To penalize divergence from the reference policy π r ​ e ​ f \pi_{ref} without additional sampling, the KL term is approximated: 𝔻 K ​ L ( π θ | | π r ​ e ​ f ) = π r ​ e ​ f ​ ( o i | q ) π θ ​ ( o i | q ) − log π r ​ e ​ f ​ ( o i | q ) π θ ​ ( o i | q ) − 1 . \mathbb{D}_{KL}(\pi_{\theta}||\pi_{ref})=\frac{\pi_{ref}(o_{i}|q)}{\pi_{\theta}(o_{i}|q)}-\log\frac{\pi_{ref}(o_{i}|q)}{\pi_{\theta}(o_{i}|q)}-1.

By replacing the value function with group-relative rewards, this implementation reduces computational overhead and memory usage compared to standard PPO.

#### A.1.3 Analogy Between ES Population Size and GRPO Rollout Count

Both GRPO and ES rely on creating different responses and then updating the model parameters via the fitness of those responses. The following section describes why the population size in ES and number of rollouts in GRPO play an analogous role in controlling parameter updates.

Following the algorithm described by Qiu et al. (2025) , an ES training update comprises of N N different seeds used to generate perturbations to the baseline model, resulting in N N different population members. Each population member is sampled with temperature at 0 to generate N N different responses, which are evaluated by a reward function to determine their fitness, which is converted into a z-score to weight the contribution of that respective perturbation to the baseline model. Further explanation can be found in A.1 .

Similarly, a GRPO training update samples N N candidate outputs from the current policy, evaluates them to obtain relative reward signals, and updates the policy via a policy-gradient objective while constraining deviation from a fixed reference policy through KL regularization ( DeepSeek-AI et al., 2025 ) . Crucially, although ES simultaneously maintains multiple different versions of a model and GRPO maintains one, ES population size and GRPO number of rollouts both determine the number of samples used to estimate a stochastic update and to form a stochastic gradient or gradient-free estimator that drives the parameter update.

### A.2 Implementation Details

Our implementations for both GRPO and ES model training and analysis is attached to this submission.

#### A.2.1 GRPO

The GRPO setup in this study is implemented on the VERL library, which employs the HybridFlow engine proposed by Sheng et al. (2025) . Training was conducted on NVIDIA RTX A6000 GPUs and the Fully Sharded Data Parallel (FSDP) protocol was used to train across GPUs. Across all experiments, we maintained 30 rollouts for GRPO to mimic the 30 mutations generated by the original ES study by Qiu et al. (2025) . To benchmark-finetuning, we used a batch-size of 200 examples along with a mini-batch size of 32 examples. A KL-Loss coefficient of β = 0.001 \beta=0.001 was used. The trainer was set to run for a total of 500 epochs, although once the validation accuracy appeared to plateau, we stopped training prematurely.

#### A.2.2 ES

We replicated the original author’s implementation of ES with two improvements: the authors found that using fp16 instead of bf16 improved validation accuracy on certain tasks. Additionally, the application of the Qwen chat template to the original task prompts improved validation accuracy on the experiment replica Countdown task for Qwen2.5-1.5B, but left model performance on all other regimes virtually the same. Runs were performed both with and without the chat template to assess the effect.

#### A.2.3 Reward functions

For the countdown task, we employ the same reward function used by Qiu et al. (2025) , adapted to fit the VERL API. An answer reward is calculated, which assigns a reward of 1.0 1.0 if the model’s answer uses all numbers once and evaluates to the provided target, and 0.0 0.0 otherwise. A separate format score is calculated, which serves to ensure that the model’s response obeys an XML-style format with <think>...</think> thinking tokens first followed by response tokens <answer>...</answer> . We take a weighted average of the two rewards to calculate the final reward to assign to the model: Reward = 0.1 ⋅ Format ​ Reward + 0.9 ⋅ Answer ​ Reward \mathrm{Reward}=0.1\cdot\mathrm{Format\ Reward}+0.9\cdot\mathrm{Answer\ Reward}

For the GSM8K, MATH, and OlympiadBench benchmarks, we employ a rule-based reward function using a binary evaluation logic. An answer reward is calculated by extracting the model’s conclusion from the final 300 characters of the response using a regex pattern. The function first identifies the #### [number] format, falling back to \boxed{...} tags if necessary, and assigns a reward of 1.0 1.0 if the extraction matches the ground truth and 0.0 0.0 otherwise.

### A.3 Hyperparameter Values

### A.4 Additional Experiments

#### A.4.1 Catastrophic Forgetting and KL

Shenfeld et al. (2025) had previously established a negative correlation between KL-divergence and previous task score. We therefore searched whether this trend also is reflected within ES-trained models . We first looked at KL-divergence between the trained and base models A1 on the newly trained task. While ES-trained models increase in KL-divergence with subsequent training steps, this behavior was not consistent when trained with GRPO. This can be attributed to the explicit KL-regularization factor in GRPO, preventing continuous drifts from the base model.

The trends for KL divergence and accuracy continue to diverge when evaluating previously known tasks A1 . ES has a clear negative correlation between KL-divergence and old task performance. The KL-divergence between the new and base models also shows an increase over number of training iterations in ES. GRPO however continues to show no association between number of training steps and KL-divergence, as well as between KL-divergence and previous task accuracy. Therefore, KL-divergence is a less reliable indicator of catastrophic convergence across GRPO and ES.

(a) Qwen-2.5-1.5B

(b) LLaMa-3.2-1B

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .