Abstract
Evaluations should do more than measure a model's current performance; they should indicate what to fix for the next model iteration and provide a means to generate targeted post-training data. Most evaluation pipelines identify weak examples, topics, or categories but leave the underlying capability failure implicit: they indicate where a model fails, not why.
We introduce CRAFT, a method that converts any rubric-based evaluation dataset into a model-specific diagnosis of weak capabilities. CRAFT treats each grading criterion as a capability probe: it extracts a capability description from every prompt-rubric pair, clusters these descriptions into a hierarchical capability tree, scores the target model at every node, and dynamically selects low-performing nodes across tree levels at the granularity where each failure is clearest. The selected weak capabilities then direct the generation of targeted supervised fine-tuning data.
Holding the data generation, fine-tuning, and evaluation setup fixed, we compare CRAFT against prompt-level EvalTree clustering and untargeted random generation on four open-source models, two professional domains (finance and legal), and 13 held-out benchmarks disjoint from the diagnostic data. CRAFT achieves the strongest finance domain average for all four models under repeated temperature decoding; in the legal domain, it is strongest for three of four models and remains within the decoding variance bands of the best baseline on the fourth.
Diagnosing weaknesses at the level of rubric criteria, rather than prompts or categories, thus yields both a sharper picture of what a model cannot do and measurably better models after fine-tuning on that diagnosis.
Blogger's Review: The CRAFT method integrates evaluation criteria with capability probing, breaking the limitations of traditional assessments. By constructing a hierarchical capability tree, CRAFT not only clarifies model weaknesses but also provides robust data support for subsequent optimization, advancing the development and application of LLMs.