Kolab is a newly released dataset containing over 115 hours of screen recordings from 50 professionals across 40 occupations collaborating with AI agents on realistic long‑horizon problems.
In long‑horizon human‑agent tasks, users place more value on ongoing interaction (e.g., plan revisions) than on an agent’s ability to finish the task in a single shot. Most existing agentic benchmarks focus on automation, so strong scores do not guarantee effective real‑world collaboration.
Kolab provides not only the full collaboration recordings but also participants’ judgments on collaboration process and output quality. Analysis reveals diverse patterns: professionals coordinate with agents at different moments and in varied ways, and the timing and manner of adopting agent suggestions significantly affect workflow efficiency.
A collaborative agent must therefore do two things that automation benchmarks never test: (1) anticipate whether an expert will adopt, modify, or reject its contribution after the chat window is closed, guiding its next steps; (2) assess collaboration quality from the evolving workflow without constant expert feedback.
The benchmark operationalizes these requirements through two tasks:
- Forecast how professionals will use agent contributions at various workflow stages;
- Verify their judgments of collaboration satisfaction and output quality.
Findings show that frontier models—both agentic and non‑agentic—overestimate successful collaboration with experts and struggle to distinguish high‑ from low‑quality interactions. Providing longer workflow context does not consistently improve prediction performance.
Consequently, current models still cannot grasp long‑horizon human‑agent collaboration. Kolab offers a valuable resource for studying collaboration grounded in real professional workflows.
Review