Introduction
Human motion capture technology is crucial for applications in virtual reality (VR) and augmented reality (AR), and head-mounted devices (HMDs) provide a scalable way to acquire real-world human motion and interaction data. Existing methods focus on two tracking paradigms: egocentric tracking, which estimates the motion of the subject wearing the device, and exocentric tracking, which captures the movements of people in the wearer's surroundings. So far, these paradigms have largely been explored in isolation.
Method
In this paper, we propose a novel distributed framework called EgoExoMoCap that jointly leverages ego- and exocentric multi-modal signals for human motion estimation. Unlike traditional motion capture systems that require bulky multi-camera setups or obtrusive mocap suits, our approach only needs two or more people wearing a pair of smart glasses. The method utilizes head (and potentially wrist) tracking signals for accurate estimation of global motion in the 3D world and combines context-aware image features based on DINOv3 to achieve robustness against noise and occlusions.
Experimental Results
Extensive experiments on two in-the-wild datasets show that our approach can robustly reconstruct motion even in challenging scenarios.
Blogger's Review: EgoExoMoCap presents an innovative approach by integrating multi-modal signals, significantly simplifying the complexity of motion capture, which could drive further advancements in the VR/AR domain. Its adaptability and robustness offer extensive prospects for future applications.