Abstract
This paper explores the possibility of merging large language models with significantly different parameter spaces through direct weighted averaging, without training or semantic alignment. Existing heterogeneous fusion methods typically involve distillation, adapters, learned latent spaces, routing, or feature alignment, while we propose a simpler recipe. We revisit this counterintuitive question through training-free dimensional adaptation followed by ratio-controlled interpolation.
In union-style merging, we expand the smaller model into the larger parameter space; in intersection-style merging, we truncate the larger model into the smaller parameter space. Across Qwen-family model pairs and benchmarks covering mathematical reasoning, code generation, language understanding, commonsense reasoning, knowledge, and instruction following, deterministic expansion largely preserves the source model function, and small-ratio interpolation can improve upon strong source checkpoints by transferring complementary capabilities.
However, near-balanced interpolation often collapses, and task-level results reveal a seesaw effect where gains on some capabilities coexist with regressions on others. These results show that simple parameter averaging, when paired with lightweight dimensional adaptation and carefully controlled ratios, serves as a surprisingly strong baseline for heterogeneous LLM merging, suggesting that the limits of direct weighted fusion may also bound what more complex heterogeneous merging methods can achieve at scale.
Blogger's Review: The methods presented in this paper demonstrate promising potential for merging heterogeneous models, achieving significant results while avoiding complex processes. The use of simple weighted averages alongside dimensional adjustments warrants further exploration and application.