Abstract
Existing human-object interaction (HOI) video generation methods are largely limited to offline short-video generation with complex driving conditions, making them unsuitable for real-time interactive applications. We present StreamHOI, a low-latency streaming framework for long-duration HOI video generation.
Key Technology
The innovation of StreamHOI lies in how to organize historical memory to preserve interactions under bounded latency. We find that the standard sink-local memory design faces a trade-off in streaming HOI generation, and different transformer blocks show different historical-memory preferences for HOI regions and surrounding areas.
To address this, StreamHOI performs offline HOI-aware block profiling and applies bias-guided memory-specialized training to adapt the generator to block-specific memory layouts. Additionally, we introduce a memory distance scaling module to enhance long-range access to early interaction states.
Performance
Extensive comparisons with both long-video baselines and recent HOI generation methods demonstrate that StreamHOI achieves strong interaction plausibility, object fidelity, human quality, and efficiency, reaching 17.6 FPS with 0.75s first-chunk latency.
Blogger's Review: StreamHOI significantly enhances interaction quality and generation efficiency in streaming HOI video generation by optimizing memory organization and module adaptation. Its innovative methodology provides new insights for real-time video generation, making it a valuable subject for further research and application.