NeFut Logo NeFut
Admin Login

[CS.AI] Total Cost of Agency: Exact Attribution of Memory Injection Cost in Multi-Agent LLM Workflows

Published at: 2026-09-23 22:00 Last updated: 2026-09-24 00:40
#optimization #LLM #Artificial Intelligence

In multi‑agent large language model (LLM) workflows each node retrieves context from memory and injects it into its prompt; the injected tokens are billed at the same per‑token rate as system prompts and user queries. Existing observability tools only report total token usage and cannot separate tokens a node generates from those it receives, making this cost component invisible to payers. This paper introduces the Total Cost of Agency (TCA) framework, decomposing workflow cost into base prompt, inference, memory injection, miss penalty, and context‑accumulation components, and proposes a two‑pass, non‑billable token count that directly measures injected tokens instead of estimating them from word‑count proxies.

On a 200‑task enterprise benchmark executed against real model APIs, memory injection accounts for 13.6% of the variable cost that a compile‑time optimizer can act on and about 12% of the full billed cost. Its share rises from a structural zero at depth 1 to 27.6% at depth 6. Injected tokens grow linearly with depth (R²=0.9974); a quadratic fit yields a negative leading coefficient, indicating no convex growth within the measured range. Reducing the retrieval window from 32 to 2 entries lowers injected tokens by 28.7% while accuracy changes remain within seed‑level variation.

The experiments also show that graph‑rewriting transforms are approximately cost‑neutral in isolation, two of the five decomposition terms are zero by construction in this harness, and total workflow cost is dominated by model‑tier assignment, which we hold fixed as prior work. Prompt caching is not evaluated; all figures pertain to the uncached case.

Review

Original Source: https://arxiv.org/abs/2609.23790

[h] Back to Home