Abstract
Model merging is promoted as a substitute for joint multi-task training, yet in the reinforcement-learning setting, this substitution is essentially never tested against the baseline it claims to replace: methods merge independently released agents precisely because a joint model is unavailable. We build the missing comparison.
Training difficulty-1 and difficulty-2 Qwen3-8B specialists on the AppWorld agent benchmark with LOOP, we merge them (TIES, RAM+) and pit the result against a jointly trained model on the same data. On task-goal completion, merging matches joint RL -- and every merge variant is statistically indistinguishable.
To explain why merge method does not matter here, we measure the geometry of the specialists' task vectors, which carries no task-sampling noise: they are near-orthogonal (cosine 0.06 - 0.10) despite ~65% support overlap, a small, shared direction that grows over training and that we calibrate against a random-init floor and a same-run ceiling to confirm it reflects learning, not the low-rank parameterization.
Because direction and support are decoupled, support and sign-based merging (RAM, TIES) collapse to near-uniform averaging. We release all code and statistics.
Blogger's Review: This paper reveals the effectiveness of model merging in multi-task reinforcement learning through geometric analysis, emphasizing the significance of task vector geometry in understanding merging strategies. While merging yields results similar to joint training, researchers should focus on the selection of merging strategies and their potential impact on training outcomes.