Abstract
On-device LLM inference faces a trilemma of response latency, limited hardware resources, and user privacy. Full cloud inference delivers strong computing power but exposes user prompts and dialogue data, while standalone on-device inference is unfeasible for most consumer and embedded edge devices. This paper presents a privacy-centric edge-cloud collaborative LLM inference framework built on endpoint-authenticated KV cache.
Framework Design
Local endpoints handle input preprocessing, embedding computation, adaptive feature optimization, KV cache authentication, speculative decoding, and low-dimensional model head calculation, while the cloud conducts authenticated decoder inference, KV cache management, token verification, and high-dimensional vocabulary projection. Endpoints fuse partial outputs, apply language-adaptive masking, and sample target tokens.
Data Privacy Protection
All transmitted data and truncated logits are quantized and AES-GCM encrypted for privacy, with core lightweight modules, draft parameters, and cache access policies kept local to avoid leakage. The framework supports heterogeneous devices including CPU-only, GPU-equipped, and embedded devices via optimized streaming, batching, and quantized ONNX deployment.
Performance Evaluation
Evaluations demonstrate that the framework reduces per-token latency by up to 46.1% and downlink payloads by up to 67.4% over baseline split inference, retaining comparable performance to full cloud inference.
Blogger's Review: This framework effectively combines edge and cloud computing to address the trade-off between privacy and performance, showcasing significant application potential, especially in scenarios with high data protection requirements. Its optimized streaming and quantization techniques offer new insights for the widespread adoption of LLMs.