In the realm of AI research, interactive simulators have emerged as essential tools for training embodied agents and generating synthetic visual data. However, existing photorealistic simulators suffer from limitations in generality, programmability, and rendering speed. To address these challenges, we introduce SPEAR: a simulator designed for photorealistic embodied AI research.
At its core, SPEAR is a Python library that connects to and programmatically controls any Unreal Engine (UE) application through a modular plugin architecture. SPEAR exposes over 14K unique UE functions to Python, representing an order-of-magnitude increase in programmable functionality compared to existing UE-based simulators. Furthermore, a single SPEAR instance can render 1920x1080 photorealistic images directly into a user's NumPy array at 73 frames per second—an order of magnitude faster than existing UE plugins—while also providing ground truth image modalities not available in other UE-based simulators (e.g., non-diffuse intrinsic image decomposition, material IDs, and physically based shading parameters).
Finally, SPEAR introduces an expressive high-level programming model that allows users to specify complex graphs of UE work with arbitrary data dependencies among work items, executing these graphs deterministically within a single UE frame. We demonstrate SPEAR's utility through diverse applications: controlling multiple embodied agents with distinct action spaces (e.g., humans, cars, robots) across various real-world UE projects; rendering photorealistic city-scale environments; manipulating UE's procedural content generation systems; rendering synchronized multi-view images of detailed human faces; coordinating interactive co-simulation with the MuJoCo physics simulator; and editing scenes using natural language via an AI coding assistant.
Blogger's Review: The launch of SPEAR marks a significant advancement in the field of embodied AI research. Its powerful programmability and efficient rendering capabilities will greatly enhance the development of applications, particularly in controlling agents and simulating interactions in complex environments. It provides researchers with a more flexible toolkit to achieve higher-level AI experiments and applications.