Ego2World is a benchmark that compiles annotated egocentric cooking videos into executable planning environments under partial observability. The compiler maps each video step and involved objects to symbolic action rules, persistent world states, and explicit task conditions, allowing researchers to feed an agent’s proposed actions into the system and verify the outcomes.
The framework separates world state from the agent’s belief, enabling controlled studies of information reuse and planning strategies across continuous tasks. We evaluated six planners on 105 cooking tasks and found that, although most operations were accepted, many task goals remained unmet. Execution traces together with condition checks distinguish interrupted runs, partial attainment, and complete executions that still fail to achieve the goal.
In a paired study using Qwen‑Plus as the language model, adding a persistent belief improved action validity by 4.15 percentage points and reduced visual‑query attempts by 90.27 %, at the cost of higher token usage and without a detectable gain in task completion.
Ego2World provides a reusable testbed for tracing how planning and memory choices affect execution, observation demand, and task attainment, bridging recorded human activity with the development and evaluation of interactive agents.
Review