NeFut Logo NeFut
Admin Login

[CS.AI] SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference

Published at: 2026-08-15 22:00 Last updated: 2026-08-16 07:03
#AI #Large Language Models #Distributed Edge Cloud Inference

We present a distributed inference framework, SPADE, that integrates speculative decoding (SD) across edge and cloud. A compact draft model deployed on the edge generates candidate tokens rapidly, and a large verifier model on the cloud validates these tokens in parallel. Accepted tokens are retained, while only rejections trigger verifier correction, substantially reducing the number of cloud queries. Our plug-and-play design shifts the bulk of computation to the edge, significantly lowers inference time and cloud cost, and preserves the accuracy of the big model without any retraining requirement. Experimental results across multiple Natural Language Processing tasks using SpecBench and CNN/Dailymail datasets demonstrate that SPADE reduces the cloud model calls by $76%$ with zero loss in accuracy as compared to the full model. Blogger's Review: SPADE provides a practical solution for deploying large language models on edge cloud, leveraging speculative decoding and distributed inference to reduce latency and cloud costs while maintaining accuracy.

Original Source: https://arxiv.org/abs/2608.13076

[h] Back to Home