Competition and Attraction Improve Model Fusion — part 1/3
Cleaned-text code points [0, 15841). Bibliography omitted here; source.md includes it.
# Competition and Attraction Improve Model Fusion
###### Abstract.
Model merging is a powerful technique for integrating the specialized knowledge of multiple machine learning models into a single model. However, existing methods require manually partitioning model parameters into fixed groups for merging, which restricts the exploration of potential combinations and limits performance. To overcome these limitations, we propose Model Merging of Natural Niches (M2N2), an evolutionary algorithm with three key features: (1) dynamic adjustment of merging boundaries to progressively explore a broader range of parameter combinations; (2) a diversity preservation mechanism inspired by the competition for resources in nature, to maintain a population of diverse, high-performing models that are particularly well-suited for merging; and (3) a heuristic-based attraction metric to identify the most promising pairs of models for fusion. Our experimental results demonstrate, for the first time, that model merging can be used to evolve models entirely from scratch . Specifically, we apply M2N2 to evolve MNIST classifiers from scratch and achieve performance comparable to CMA-ES, while being computationally more efficient. Furthermore, M2N2 scales to merge specialized language and image generation models, achieving state-of-the-art performance. Notably, it preserves crucial model capabilities beyond those explicitly optimized by the fitness function, highlighting its robustness and versatility. Our code is available at https://github.com/SakanaAI/natural_niches .
## 1. Introduction
Open-source generative models have enabled the proliferation of thousands of specialized variants, fine-tuned by practitioners to meet their specific needs. In an environment where diverse models are freely accessible, the ability to merge and consolidate this wealth of knowledge into a single model becomes increasingly valuable. This process, known as model merging ( Labonne, 2024 ) , has gained traction, as evidenced by the widespread presence of merged models on the Open LLM Leaderboard ( HuggingFace, 2023 ) .
Model merging initially relied on manually adjusting coefficients to combine seed models, a process guided by intuition and requiring significant trial and error to optimize performance for specific tasks. Recently, evolutionary algorithms have streamlined this process by automatically searching for optimal coefficients ( Akiba et al., 2024 ; Kuroki et al., 2024 ) , significantly improving merging efficiency. However, one manual step remains: developers must group model parameters into fixed sets before merging, restricting the search space for potential combinations (see Figure 1 left). To address this limitation, we propose Model Merging of Natural Niches (M2N2), an evolutionary algorithm with three key features:
(1) Evolving the Merging Boundaries . Existing methods partition the parameters of each seed model into fixed groups (e.g., layers) and optimize merging coefficients within these predefined boundaries, limiting the scope of exploration. In contrast, M2N2 iteratively merges two models at a time, using flexible split points to divide parameters. Instead of working with static models, we maintain an evolving archive of models. As the number of generations increases, M2N2 progressively explores a broader set of boundaries and coefficients (see Figure 1 right), enabling increasingly complex combinations when beneficial. This gradual, optional increase in complexity ensures a more extensive search while maintaining computational efficiency.
(2) Managing Diversity . Model merging is most effective when combining diverse models, making diversity preservation essential. However, the challenge lies in defining which characteristics should remain diverse. Many approaches require manually specified diversity metrics, however, we believe it is becoming increasingly hard to come up with efficient diversity metrics as models and tasks grow in complexity. Instead, we incentivize diversity of high-performing models as Nature does — through competition for limited resources.
(3) Attraction . We introduce a heuristic for pairing models based on their complementary strengths, which improves both efficiency and the final model performance. Mate selection remains an underexplored aspect of genetic algorithms, yet it becomes increasingly crucial as the computational costs of crossovers (merging) grow. M2N2 highlights the importance of this factor and encourages further research in this area.
Our achievements and key contributions are summarized below. • We introduce M2N2, a novel evolutionary approach with three key components – competition, attraction, and model fusion with split points. Through comprehensive ablation studies, we demonstrate that these components significantly improve model merging and could enhance other evolutionary algorithms using crossover operations.
• We present the first application of model merging for training models from scratch, showing that our method surpasses existing evolutionary algorithms in both performance and computational efficiency.
• We scale M2N2 to Large Language Models (LLMs) and diffusion-based image generation models, highlighting key benefits of gradient-free optimization, including stable model fusion without catastrophic forgetting, compatibility across models trained on different objectives, reduced memory footprint by avoiding gradient computations, and preservation of model capabilities without requiring access to the original training data.
## 2. Related Work
In this section, we review two relevant areas of research: the emerging field of model merging, and the well-established diversity preservation mechanisms in genetic algorithms. While model merging represents a novel approach that has evolved rapidly from manual to automated techniques, diversity preservation in evolutionary algorithms has been extensively studied over several decades.
### 2.1. Model Merging
Model merging introduces an innovative approach for integrating the strengths of multiple pre-trained models. In contrast to fine-tuning, which focuses on refining a single pre-trained model, model merging can leverage several models concurrently without requiring back-propagation. This has allowed the method to combine extremely large models for tasks involving subjective goals, like customizing an image generation model to reflect personal tastes. Notably, the release of Stable Diffusion (SD) ( Rombach et al., 2022 ) and open-source interfaces ( AUTOMATIC1111, 2022 ) enabled practitioners to merge different SD fine-tunes manually, using techniques like linear and spherical linear interpolation (SLERP) ( White, 2016 ) . These early efforts demonstrated the potential of model merging in combining specialized capabilities into a single unified model. Curiously, this modern trial-and-error search for personally appealing images seems extremely inline with earlier evolutionary art systems like Picbreeder ( Secretan et al., 2008 ) , where users could combine and evolve neural networks to create images that matched their subjective preferences through an exploratory interface.
Subsequent research has approached the model merging problem from two complementary directions: minimizing interference between models and automating the merging process. Methods such as TIES ( Yadav et al., 2023 ) and DARE ( Yu et al., 2024 ) introduced strategies to balance the contributions of individual models while minimizing interference, ensuring that the strengths of each model are preserved without mutual disruption. They also expanded the applications of model merging to the natural language domain.
Evolutionary algorithms like CMA-ES ( Hansen and Ostermeier, 2001 ) were later applied to automate the search for optimal merging coefficients. As explored in ( Akiba et al., 2024 ) , these methods not only automate what was previously a manual, iterative process but also significantly improve efficiency.
While previous research centered on merging pre-trained models, we show that merging can efficiently be used to evolve models from scratch. Additionally, unlike earlier methods that required manual partitioning of model parameters, we automate and optimize this process during the evolutionary process.
### 2.2. Overview of Diversity Preservation in Genetic Algorithms
Diversity preservation in Genetic Algorithms (GA) is crucial for finding multiple solutions to multimodal problems ( Wong, 2015 ) and to prevent premature convergence. We believe this is particularly important when using crossover operations (such as model merging). These operations benefit from diversity while at the same time reducing it, which may lead to premature convergence if not counter-acted by a diversity increasing mechanism. In this section, we provide a quick overview of the two main methods for diversity preservation in GA: 1) crowding ( De Jong, 1975 ; Wong et al., 2012 ) and 2) fitness sharing ( Goldberg et al., 1987 ; Deb and Goldberg, 1989 ; Goldberg et al., 1992 ; Pétrowski, 1996 ) .
Crowding methods involve first applying mutation and crossover to produce new candidate solutions. These candidates then compete for inclusion in the population, but only against other candidates that are similar, based on a predefined criteria such as genetic or phenotypic distances. This selective competition helps maintain diversity within the population by preventing any single solution type from becoming overly dominant. A similar mechanism for selective competition is used in the popular algorithm of MAP-Elites ( Mouret and Clune, 2015 ) . In MAP-Elites, the solution space is divided into a multidimensional grid, with each cell representing a species defined by one or more predefined behavior descriptors. New candidates are placed into cells based on their descriptors and replace existing solutions only if they perform better. The real challenge of this method lies in defining descriptors that promote the desired type of diversity.
Fitness sharing requires each individual to share its rewards with others. In explicit fitness sharing, the researcher defines a distance function that is used to cluster similar individuals into a species, each individual then shares its fitness with other members of its species, making it more difficult for any single species to grow excessively large. A notable example is the NEAT ( Stanley and Miikkulainen, 2002 ) algorithm, known for evolving neural networks topologies, which clusters solutions into species by measuring genotypic differences (distances in network topologies). Implicit fitness sharing ( Smith et al., 1993 ; Darwen and Yao, 1996 ) , is seen as the more natural method because, as in Nature, it protects niches rather than species. A niche is a group of individuals that compete for the same resources, while a species is defined as group that can interbreed and typically have small genetic and phenotypic differences. Usually, members of the same species compete for the same resources (e.g., food, partners, shelter), but vastly different species can also compete for the same vital resources like nesting sites or food sources (e.g., birds and bats, lions and hyenas). Implicit sharing does not rely on custom distance metrics. Instead, it simulates natural competition for limited resources, promoting diversity as individuals who can derive their fitness from less contested resources are favored. While this approach has received less attention recently, we argue it deserves renewed consideration. The high-dimensional nature of modern AI tasks and models makes defining meaningful diversity metrics particularly challenging. By leveraging natural resource competition rather than explicit distance metrics, implicit fitness sharing offers an elegant and scalable solution for contemporary applications. We hope this work helps revitalize interest in this underappreciated approach to diversity maintenance. We provide more details in Section 3 .
## 3. Natural Niches
In model merging, the goal is to find the optimal parameters θ ∗ \theta^{*} for a merged model from a set of K K seed models, each of which is characterized by its model parameters θ i \theta_{i} ( i = 1 ⋯ K i=1\cdots K ), so that the optimization goal, normally in the form of summation or average of task scores, is maximized. The following equation expresses this description mathematically: (1) θ ∗ = arg max ∑ j = 1 N θ s ( x j ∣ θ ) , where , θ = h w ( θ 1 , ⋯ , θ K ) \theta^{*}=\arg\max_{\theta}\sum_{j=1}^{N}{s(x_{j}\mid\theta}),\text{where},\theta=h_{w}(\theta_{1},\cdots,\theta_{K}) Here, h w h_{w} is the model merging function parameterized by w w ’s that correspond to fixed model merging boundaries (e.g., one scalar w k , l w_{k,l} for the l l -th layer in the k k -th seed model), s s is the score function for a certain task, x j x_{j} is a task example, and N N is the number of examples to be evaluated. In Natural Niches (M2N2), we propose modifications to the merging function h h to allow the evolution of the merging boundaries, and adjustments to the optimization goal to promote diverse solutions.
### 3.1. Eliminating Fixed Model Merging Boundaries
In the formulation above, finding θ ∗ \theta^{*} boils down to searching for the optimal model merging parameters w w in h w h_{w} . To get rid of the constraints of fixed model merging boundaries and thus allow more flexibility, we propose to include these boundaries together with the mixing parameters into the evolutionary process. Concretely, M2N2 maintains an archive of models, which is initialized with the K K seed models. At each training step, M2N2 randomly picks two models A A and B B from the archive, and samples two parameters ( w m , w s ) (w_{m},w_{s}) that determines the mixing ratio and the split-point in the models’ parameters space. It then merges models A A and B B with the following formula, and inserts the new model into the archive if it outperforms the worst individual. (2) h M2N2 ( θ A , θ B , w m , w s ) = concat ( CLOSE \displaystyle h_{\mathrm{M2N2}}(\theta_{A},\theta_{B},w_{m},w_{s})=\text{concat}\big( f w m ( θ A < w s , θ B < w s ) , \displaystyle f_{w_{m}}(\theta_{A}^{<w_{s}},\theta_{B}^{<w_{s}}), OPEN f 1 − w m ( θ A ≥ w s , θ B ≥ w s ) ) \displaystyle f_{1-w_{m}}(\theta_{A}^{\geq w_{s}},\theta_{B}^{\geq w_{s}})\big) Here, θ < w s \theta^{<w_{s}} and θ ≥ w s \theta^{\geq w_{s}} indicate the sub-arrays of model parameters before and after the split-point indexed by w s w_{s} . f t ( θ A , θ B ) f_{t}(\theta_{A},\theta_{B}) is a spherical linear interpolation of rotations (SLERP) function that interpolates ( θ A , θ B ) (\theta_{A},\theta_{B}) with t t . As shown in the right part of Figure 1 , our method incrementally expands the search space by exploring a broader set of boundaries and coefficients. This gradual introduction of complexity ensures a wider range of possibilities while maintaining computational tractability.
### 3.2. Encouraging Diversity via a Modified Optimization Goal
Competing for limited resources naturally promotes diversity, favoring individuals who can tap into less contested resources. In the context of the optimization goal in Equation 1 , where a sum of scores from all the examples is being maximised, each score is a “resource” that contributes to the fitness of a solution. By limiting the resource supply, M2N2 sparks competition which naturally favors individuals that take over new niches. Concretely, we limit the total fitness a population can extract from a data point x j x_{j} by a capacity c j c_{j} . The amount of fitness a candidate solution derives from a data point is proportional to its score relative to the aggregate score of the population. Our modified goal becomes: