LLM agents retrieve skills from public hubs that can store up to about 100,000 entries and serve many models. As models improve, a hub must identify which skills a model has already internalized—i.e., the model can perform the associated task without the skill—and retire those skills for that model. Existing internalization metrics are unreliable. Published methods compare runs with and without a skill using the gain metric, but gain is near zero both when a skill is internalized and when neither the model nor the skill provides the capability; we call this situation "uncovered."
We instead withhold the skill, ask the model to solve a task synthesized from the skill, and record the unaided score. A high unaided score is trusted only when another model shows a clear gain from the skill, because nearly half of high scores arise from tasks that benefit no model. Experiments on four models show that applying a published gain rule with one task per skill removes over half of the uncovered skills. On SkillsBench, 28 of 31 retirements rely on probes that no run passed in full, while retiring based on the unaided score incurs no detectable reward loss. Unassisted verdicts reproduce across tasks more often than the gain rule (Cohen's κ = 0.42 vs 0.23) but differ across models, so a hub must read them per model.
We release 1,251 skills with their tasks and item scores under four models for the community.
Review