LLM‑based software systems increasingly rely on evaluations as quality gates throughout the development lifecycle. Prior work often tackles only a single facet of evaluation reliability, leaving many practical requirements unmet. This paper introduces an end‑to‑end eval pipeline that first creates a checklist and then applies a learned aggregation model to the checklist responses. The aggregation model is trained to fuse answers from multiple LLM judges, boosting inter‑judge agreement and improving accuracy against human annotations. The pipeline also supplies self‑consistency checks, explanation generation, and prediction uncertainty estimates. Empirical results show that the full system outperforms baselines in consistency, interpretability, and uncertainty quantification.
Review