Competition and Attraction Improve Model Fusion — part 3/3

Cleaned-text code points [31219, 38101). Bibliography omitted here; source.md includes it.

Training and Testing metrics : During training, we give the Japanese captions as prompts to the model and calculate the cosine similarity between the CLIP features of the correspondent images from COCO and the generated ones. This metric is normalized to fall within the range of 0 to 1, which we refer to the Normalized CLIP Similarity (NCS) metric. While models generally achieve high absolute NCS values, small differences can significantly impact performance. To emphasize these relative differences, we adjust the scores before computing the fitness score: for each training sample, we subtract the corresponding worst score achieved by the population. This adjustment emphasizes relative performance, increasing the competition for training samples and favoring more diversity in the population’s skills.

Baseline : Our work builds on ( Yoshikawa et al., 2017 ) approach using CMA-ES for merging. While we use the same dataset, our method maximizes NCS instead of minimizing the Fréchet Inception Distance (FID) metric ( Heusel et al., 2017 ) as they did. This choice enables per-sample evaluation necessary for resource competition, whereas FID only provides aggregate performance across the training set.

Qualitative Results

Diversity and Visual Capabilities : Figure 6 illustrates how our merged model successfully combines the strengths of individual seed models while mitigating their weaknesses. Note that if we exclude our merged model, we observe that each seed model produced both the highest and lowest quality outputs for different test cases. Additionally, it is extremely hard to find a clear pattern that describes the speciality of each model or inform us on how to create an effective and custom diversity metric. Our diversity preservation mechanism addresses this challenge by automatically preserving the models that uniquely excel on training samples where other models underperform.

The merged model demonstrates two key improvements over the seed models. First, it generates more photorealistic images, which aligns with our training set of real photographs. Second, it shows enhanced semantic understanding of the input captions. For instance, in the rightmost column of Figure 6 , while several seed models generated visually appealing bikes, our merged model not only specifically focused on capturing the bike’s registration number display area as specified in the caption, but also produced an image that looks like an actual photograph rather than a synthetic rendering.

Language Understanding : Figure 7 shows that our model has a good understanding of both Japanese and English, despite being evolved exclusively with Japanese captions! This emergent bilingual ability exemplifies a key advantage of model merging: it enables the aggregation of complementary capabilities while avoiding the catastrophic forgetting typically associated with gradient-based training methods. Note that the seed models show very distinct language capabilities: JSDXL performs better with Japanese prompts while the other models perform better with English prompts.

Quantitative Results

Performance : Table 2 demonstrates that our model achieves a superior NCS score on the test set compared to all other models. Additionally, we surpass the model merging baseline, CMA-ES, in FID performance, even though this baseline was explicitly trained to minimize FID on the training set.

Language Understanding : We evaluated the models’ ability to maintain semantic consistency across languages by generating pairs of images: one from Japanese captions (sampled from our test set) and another from their GPT-4o translated English versions. For each image pair, we computed the cosine similarity between their CLIP feature representations, which captures how semantically similar the generated images are. Averaging these similarities across 100 diverse caption pairs, Table 3 demonstrates that our model achieves significantly better cross-lingual consistency than the other models. This statistically confirms what we had already observed in our qualitative results.

## 5. Limitations & Future Work

The feasibility of model merging strongly depends on the degree of similarity between models. As demonstrated in ( Yu et al., 2024 ) , when fine-tuned models deviate significantly from their base models—often due to extensive, divergent training—merging becomes impractical. We hypothesize that models with divergent state representations are incompatible for merging. However, a standardized metric for model compatibility has yet to be established. Defining such a metric could allow it to be used as a form of regularization during preprocessing (e.g., fine-tuning), enabling better control over model compatibility and ensuring the success of merging.

We believe there is a strong evolutionary pressure for models that are co-evolving together to remain compatible for merging. Should one model, diverge and become incompatible with others, it would no longer produce viable offspring, halting its improvement and leading to its eventual extinction. Testing this hypothesis through further research would provide valuable insights into the dynamics of model co-evolution. Moreover, incorporating a compatibility metric into the attraction heuristic could facilitate the co-evolution of distinct species of models, defined as groups that merge with one another but not with others.

## 6. Conclusion

In this paper, we present the first application of model merging for training models from scratch and demonstrate that it achieves top-tier performance and efficiency when combined with a diversity-preservation technique. Furthermore, this approach scales effectively to LLMs and diffusion-based image generation models. Our ablation studies reveal that the proposed mechanisms of competition, attraction and the use of split-points significantly enhance the performance of model merging and have the potential to benefit other evolutionary algorithms utilizing crossover operations.

Our LLM merging experiments highlight the ability to combine vastly different skills, enabling multi-task capabilities without requiring access to the original training data. Remarkably, in diffusion model experiments, we observe that the merged models retain English language capabilities despite being optimized exclusively for Japanese tasks. This finding underscores the promise of model merging as a robust transfer learning mechanism that resists catastrophic forgetting, which is typically encountered in fine-tuning.

Finally, we hope this work revitalizes interest in two underexplored areas: mate selection algorithms and implicit fitness sharing as a mechanism for diversity preservation. These concepts are increasingly critical as crossover operations, such as model merging, become computationally more expensive, and as models and tasks grow in complexity.