Abstract
Vision-Language Models (VLMs) serve as the perceptual backbone of embodied AI, yet their energy consumption on edge hardware remains poorly understood. Current efficiency efforts mainly focus on reducing visual tokens, implicitly assuming that visual processing is the primary energy cost. We overturn this assumption through the first systematic energy profiling of on-device VLM inference, encompassing five models across three architecture families, four input resolutions, and two hardware platforms (NVIDIA RTX 3070 and Jetson Orin NX).
Key Findings
-
Model-Intrinsic Constant: Average inference power is a model-intrinsic constant, invariant to input resolution, image complexity, and prompt type, with less than 5% variation across all conditions. This indicates that energy variation across inputs must arise from inference time variation, not power draw.
-
Output Token Cost: Each output token costs 11 to 39 times more wall-clock time than each input token due to the compute-bound and memory-bound asymmetry between prefill and decode, making output token count the dominant driver of both latency and energy.
-
Image Complexity Impact: Image complexity, measured by the number of objects in an image, induces up to 4.1 times energy differences at identical resolution. This variation arises not from increased visual processing cost, but from differences in output length.
These findings expose a fundamental limitation of visual token pruning: even removing all visual tokens saves at most 10% of total energy for fixed-token models. Across models spanning 1 billion to 8 billion parameters, controlling output length saves up to 97% of total energy, with the energy dominance of decoding growing stronger at larger model scales. In short, the true energy bottleneck in edge VLM inference is not what the model sees, but how much it says.
Blogger's Review: This research uncovers deep-seated issues regarding VLM energy consumption in edge computing, highlighting the significance of outputs during inference. As AI model sizes increase, strategies to optimize output length will be crucial for enhancing energy efficiency, providing important guidance for future model designs.