Humans can easily repurpose a book, a stone, or a shoe to drive a nail, but robots trained on specific tools struggle to transfer the same functionality to novel tools. This phenomenon is formalized as functional generalization. Despite sharing a visually recognizable functional intent, the perceptual similarity does not extend to the action space, where each tool necessitates a distinct motor pattern.
To bridge this gap, we explore intermediate representations, including affordance images, human video prompts, and 2D keypoint trajectories, discovering that keypoint trajectories provide the best balance between functional expressiveness and action groundability. Building on this, we propose Functional Reasoning and Grounded Execution (FORGE), a two-stage policy that decouples functional reasoning from action execution: predicting generalizable keypoint trajectories from action-free data, then grounding them into robot actions with limited demonstrations.
In a seven-tool hitting-function benchmark, FORGE consistently outperforms state-of-the-art methods on unseen tools in both simulation and real-world settings, achieving over 2X improvement in average success rate.