Large language models (LLMs) are increasingly utilized for program-aided reasoning, agentic decision making, and structured task execution, yet these applications often incur high inference costs. We introduce MiniCache, a reusable program caching framework that transforms Program-of-Thought (PoT) programs into parameterized cache objects, enabling reusable computation across structurally similar requests.
MiniCache reuses the same small model for semantic variable extraction on cache-hit requests and for speculative drafting during target-LLM generation, thereby reducing costly target-LLM invocations while maintaining task quality. Experiments on shopping-style request datasets such as WebShop, Formula, and CodeTAT-QA demonstrate that MiniCache improves the trade-off between inference latency, cache reuse, and accuracy, achieving up to 3.1x lower latency and 2.8x higher throughput under parallel serving. These results indicate that small models are most effective not as replacements for large models but as lightweight interface models that enable reliable and efficient reusable program caching.
Blogger's Review: MiniCache highlights the significance of small models in LLM inference, emphasizing the role of efficient caching mechanisms in reducing inference costs. This approach provides a fresh perspective for future LLM applications, especially in resource-constrained environments, warranting further exploration and implementation.