Retrieval‑augmented generation (RAG) improves language‑model outputs by grounding them in external passages, but the retrieved context lengthens prompts, inflating pre‑fill work, KV‑cache size, memory traffic, latency, and energy consumption. Context compression—pruning retrieved text before generation—offers a natural remedy, yet most state‑of‑the‑art methods use a fixed compression budget set offline or applied statically at inference time. This static view ignores workload fluctuations and the live state of edge devices.
On an edge SoC, compression is not free: the compressor runs on the same chip and consumes latency and energy that can offset any savings during generation. This paper proposes a telemetry‑informed adaptive compression vision and validates it on the NVIDIA Jetson AGX Thor. Experiments employ Llama and Qwen generators, the Natural Questions and HotpotQA datasets, and the LLMLingua‑2 compressor.
Measurements reveal that for 7B‑8B models, generation dominates the per‑query budget, accounting for roughly 90% of latency and 91% of GPU energy. Varying the compression rate uncovers an adaptive operating region: mild compression may miss energy‑saving opportunities, while overly aggressive compression harms inference quality. An intermediate compression level can cut GPU energy by up to 53.2% and overall SoC energy by up to 48.2% with negligible quality loss.
The authors argue for runtime policies that dynamically adjust compression, guided by workload features (e.g., query length, model size) and edge telemetry (e.g., GPU utilization, power thresholds), to jointly optimize energy and quality.
Blogger's Review: The study rigorously quantifies the cost of on‑device compression and presents a practical adaptive strategy, offering valuable guidance for energy‑efficient deployment of RAG systems on edge hardware.