Prompt optimization for text classification encompasses diverse approaches, including demonstration selection, exploration-based search, and error-driven diagnosis, each with known but incompletely characterized strengths and limitations.
We conduct a comprehensive empirical study across various classification benchmarks (2 to 150 classes) comparing these paradigms through both quantitative evaluation and qualitative analysis of optimization traces, revealing that each paradigm excels on structurally different task types and no single method dominates.
Guided by these insights, we propose Error-Guided Optimization (ERGO), an error-driven method that iterates over the full training set in non-overlapping batches, diagnoses classification failures, and generates targeted decision rules through a diagnose-prescribe-rewrite feedback loop.
ERGO achieves the best accuracy on tasks where errors concentrate in specific confused label pairs (termed boundary-learnable tasks): TREC: 90.0%, CLINC150: 94.4%, converges in 3-5 iterations, and produces interpretable decision rules.
While ERGO does not achieve the highest overall average, it fills a complementary role: demonstration-based ICL wins on coverage-dependent tasks, exploration-based search wins on many-class intent, and ERGO wins where decision boundaries are learnable from error patterns.
We provide a complementarity framework linking task characteristics to optimal paradigm selection, offering practical guidance for practitioners.
Blogger's Review: This article provides an in-depth analysis of various prompt optimization methods and introduces the ERGO method, showcasing how to enhance text classification accuracy by leveraging error patterns. Particularly, ERGO demonstrates its unique advantages in addressing boundary-learnable tasks, making it a valuable approach for practical applications and further research.