Abstract
AI agents are enabling a new paradigm of agent-augmented real-time communication (RTC), where humans focus on high-level collaboration, while agents autonomously retrieve, analyze, and generate information in real time to support their interactions. These apps enable new experiences across various domains: for example, when corporate employees co-author a legal document, their agents can discuss and draft on their behalf, sparing them the burden of manually reviewing each other's work.
However, existing cloud-based agents suffer from privacy risks and unscalable server costs, making on-device agent-augmented RTC a promising alternative. This on-device paradigm introduces a new networking challenge: contention between concurrent traffic flows generated by humans (for live video streaming) and agents (for sending context files for analysis).
We design HFS, a framework to ensure both high live video quality and low agent response latency in agent-augmented RTC apps. We achieve this through an app-guided multi-flow transport approach, where a unified app-layer orchestrator jointly controls the sending rates of live video and agent context flows based on their heterogeneous app requirements.
Our prototype built atop WebRTC and llama.cpp demonstrates that HAFS outperforms baselines, achieving 1.5x higher video quality while reducing agent response time by 31%.
Blogger's Review: This research successfully addresses privacy and cost issues through on-device agent-augmented real-time communication, showcasing effective network traffic management capabilities. Its broad application prospects and practicality will be validated in more real-world scenarios in the future.