UpgradeBench is a longitudinal benchmark that puts model‑upgrade decisions at its core. It spans four consecutive Qwen releases, one continuation checkpoint, six downstream tasks, two model sizes, and is enriched with OLMo checkpoints whose training lineage is known. The benchmark isolates three fundamental questions: does a new checkpoint improve the performance of specialists trained with a fixed recipe, can specialization assets be transferred across versions, and what recovery resources are available.
Results show that upgrade gains vary dramatically across task‑scale‑release episodes. Some retrained baselines improve noticeably on newer releases, while others stay within training noise; durability ranges from less than one release interval for text‑to‑SQL to over fourteen months for intent classification. Direct adapter copying succeeds regardless of architecture or model family: on OLMo, retention stays between 0.88 and 0.99 after 46 B‑token continued pre‑training but drops to zero after 2.9 T tokens. Neither annealing nor model‑souping adds extra harm, and portability decays with continued‑pre‑training distance.
Given preserved input data, teacher relabeling can recover target‑base specialists without fresh gold annotations, although compute savings are not guaranteed. Simulating a fixed decision policy over 33 upgrade episodes yields a mean quality regret of 0.37 pp with zero behavioral regressions, at one‑third the compute and label cost of full retraining. A lightweight CKA probe over 256 prompts predicts cross‑version adapter portability (Spearman 0.74 across eight model pairs).
We release per‑example predictions, cost logs, split manifests, and evaluation code.
Blogger's Review: UpgradeBench offers a practical, data‑driven framework for navigating the trade‑offs of LLM specialist upgrades, especially highlighting the nuanced behavior of adapter portability and the potential compute savings, making it a valuable resource for both researchers and industry practitioners.