Fine-tuning large language models (LLMs) is compute-intensive and prone to errors, with model performance highly dependent on data quality and hyperparameter choices. Naive runs can even degrade performance, raising the practical question: can we predict fine-tuning performance before committing to a full training run?
We present TUNEAHEAD, a lightweight framework for pre-hoc prediction of fine-tuning performance. TUNEAHEAD encodes each candidate run as a meta-feature vector that combines static dataset descriptors with dynamic probe features from a short standardized probe. A predictor maps these features to performance estimates, while SHAP-based attributions provide interpretable diagnostics revealing which specific features drive the prediction.
Across 1,300+ fine-tuning runs on Qwen2.5-7B-Instruct, TUNEAHEAD consistently outperforms strong baselines like Early-Stop Extrapolation and ProxyLM. On a held-out test set of 370 runs, TUNEAHEAD achieves an RMSE of 1.47 percentage points, placing 95.1% of predictions within +3/-3 percentage points of the true score. These accurate continuous predictions support practical go/no-go screening policies, reducing unnecessary full fine-tuning while retaining the most promising runs.
Blogger's Review: The introduction of TUNEAHEAD significantly enhances the efficiency of the fine-tuning process. With its precise predictive capabilities, researchers can optimize training workflows under resource constraints, avoiding unnecessary computational waste. This framework's successful application is likely to inspire innovation and practice in model fine-tuning strategies.