In multi-task model merging, separately trained expert models are typically combined into a single model that handles all tasks without co-training. The standard practice merges experts at their optimal validation loss. We challenge this convention by systematically studying how the training duration of domain experts affects the quality of the merged model.
We fine-tuned experts on five domains (Math, Code, Instruction Following, Multilingual, and Safety) across three model sizes (Qwen 3.5 0.8B, 2B, and 4B), saving checkpoints from 25\% to 500\% of the optimal training steps and evaluating five merging methods at each duration.
Our findings reveal a striking method-dependent pattern: simple averaging degrades sharply with overfitting, while sparsification-based methods achieve their best performance well past the validation optimum. We formalize this through bias-variance decomposition analysis, drawing a parallel to random forests where averaging benefits from high-variance individual learners. These results suggest that training duration and merging method should be chosen jointly rather than independently.
Blogger's Review: This paper empirically highlights the significance of training duration in the model merging process, providing new insights for multi-task learning. It emphasizes the interplay between method selection and training strategy, which is crucial in the rapidly evolving landscape of LLMs. A well-considered model merging strategy could significantly enhance model performance.