Existing methods for testing deep neural networks (DNNs) primarily prioritize test inputs likely to reveal model faults under a fixed labeling budget. However, selecting the right budget is challenging: too little testing misses failures, while too much incurs unnecessary labeling costs. This work studies the stopping problem in DNN testing, formulating it as a cost-benefit decision process where labeling an input incurs cost $c$ and discovering a fault yields value $v$.
Based on this formulation, we introduce \textit{AdaStop}, a framework that estimates the marginal fault discovery rate during testing and stops labeling when the estimated rate falls below the threshold $\tau = c/v$. Experiments across multiple datasets, architectures, and selection strategies show that $65$–$84\%$ of faults can be discovered using only $9$–$31\%$ of the labeling budget.
Blogger's Review: AdaStop's cost-aware early stopping strategy provides an efficient solution for DNN testing by dynamically adjusting the labeling budget, significantly enhancing fault discovery efficiency. This method not only reduces testing costs but also guides future research in testing optimization. Its performance across multiple datasets demonstrates the framework's broad applicability.