Project Background
Evaluations (Evals) often become a deployment bottleneck for real-world AI applications: public benchmarks rarely match a team's users, context, or policies, and human review is often tedious to scale. Motivated by our work with AI applications in the public sector, the Kaleidoscope project addresses recurring evaluation challenges, especially when applications must satisfy local policy and governance requirements.
Kaleidoscope Workflow
We present Kaleidoscope, an integrated workflow for contextual functional evaluation that links persona-based test generation, contextualized rubrics, and human review for reliability-gated automated scoring. Generated test cases are scored against application-specific rubrics; human annotations provide reviewable labels; and LLM judges automate scoring only when their agreement with those labels meets a configured threshold.
Experiments and Results
Kaleidoscope thus serves as a practical, inspectable, iterative workflow for product teams. We report early evidence from a three-week pilot across four organizational use cases and custom-rubric judge experiments on 108 annotated Q&A pairs spanning four domains and 14 evaluation dimensions. The results highlight useful features for end-to-end reliable, automated scoring.
Blogger's Review: The Kaleidoscope project addresses a significant bottleneck in AI application evaluation by integrating human review with automated scoring, particularly in compliance with local policies. Its inspectability and iterative nature provide great convenience for product teams, making it a development to watch closely.