AI productivity is often gauged by completion time, economic value, or outcome quality, yet these metrics typically treat collaboration as a black box, recording only the output while ignoring the interaction cost required to achieve it. Inspired by economics literature, we propose a productivity‑oriented framework that evaluates human‑AI collaboration as outcome quality relative to interaction cost. Across two datasets covering four tasks we uncover four main insights:
- Sessions with identical quality ratings can differ by up to 70× in interaction cost;
- Quality‑cost relationships are task‑dependent, with some tasks benefiting from extended interaction and others favoring rapid convergence;
- Subjective user ratings are not reliable proxies for productivity;
- Productive sessions feature agents probing earlier and users spending less effort repairing the interaction.
By separating productive success from costly success, the framework makes interaction cost visible and shows how dialogue analysis can inform the evaluation and design of AI systems.
Review