How KV Caching Lowers AI Agent Costs

As an agent works through a task, its growing history of instructions, messages, and tool results is processed repeatedly. Context caching reduces repeated computation by retaining the model’s attention state in a key-value (KV) cache. Reusing this state reduces the computation needed to complete the inference request.

Figure 1 shows how four parts of the inference system work together. KV caching supports reuse during generation, and prefix caching extends it across model calls. A memory hierarchy retains more histories, while cache-aware routing directs requests to workers that can reuse them. This post explains how these mechanisms work together to lower inference costs as contexts grow and more agents run concurrently.

Reuse KV state within a request

A model generates a response one token at a time, using the prompt and previously generated tokens to predict what comes next. Panel 1 of Figure 1 shows this generation loop. Figure 2 expands a step inside it to show how retaining earlier KV state reduces repeated computation.

Each attention layer computes queries (Q), keys (K), and values (V) from token representations. The current token’s query is compared with the keys to determine how strongly to weight the corresponding values. Without KV reuse, as shown on the left, each generation step reruns the growing sequence, rebuilding earlier token representations and their keys and values.

With KV reuse, shown on the right, the engine retains the keys and values already computed for the prompt and earlier output tokens. At each layer, it computes new Q, K, and V only for the current token ti, and appends K(ti) and V(ti) to the cache. The current query then attends to the accumulated keys and values. After the model completes its computation, it samples ti+1, which becomes the input to the next step. The cache is initially populated during prefill and grows as the model processes output tokens. Keeping this state available throughout the request avoids rebuilding earlier state every time the model generates another token.

KV caching for sparse attention

KV caching makes generation more efficient by saving the keys and values computed for earlier tokens. As the history grows, the next challenge is accessing that saved state. With full attention, each generation step still reads KV across the available context, increasing attention computation and memory traffic.

Sparse attention, such as the DeepSeek Sparse Attention (DSA) used in GLM-5, addresses this problem by selectively reading from the main KV cache. On the right of Figure 3, an index-key cache is maintained alongside the main KV cache. An indexer derives a separate query from the current token’s hidden state hi and scores the available positions using those index keys. The original GLM-5 configuration selects up to 2,048 positions per query from a 128K token context. Main attention then uses its own query Qi to attend only to the KV entries retrieved from those positions. The full main KV history remains available for subsequent steps, and the index keys require additional storage. The benefit comes from reading fewer main KV entries and performing less main attention computation.

Reuse KV state across requests

The KV state computed during one model call can remain useful after that call ends. Prefix caching reuses the retained state when a later request begins with an identical token sequence under the same model and compatible serving configuration. Panel 2 of Figure 1 illustrates this reuse. After a coding agent runs a test, its next request contains the unchanged history ABC followed by the new test result D. The engine locates the cached KV blocks for ABC and processes only D during prefill. D still attends to the saved context, and its newly computed KV extends the prefix available for subsequent requests.

Figure 4 shows how the benefit accumulates as an agent takes more steps. Each additional call revisits a longer history, so the cumulative volume of repeated input grows faster than the number of calls. By 40 calls, those prefixes account for 4.3 million repeated input tokens per task. As the agent loops, the illustrated savings across 1,000 tasks increase to $516 at 40 calls per task, an 80% reduction in input token cost.

Realizing these savings requires keeping the matching KV state available when each request arrives. As histories grow and more agents run concurrently, retaining that state becomes a memory-capacity problem.

Extend KV cache capacity for concurrent agents

Cache capacity determines how much context can remain available for reuse. Longer histories require more KV state, and concurrent agents bring more histories to retain. While shared prefixes can use the same cached blocks, each history’s distinct content adds to the footprint. In GPU memory, this state also competes for space with model weights and inference workspaces.

Figure 5 shows how requirements grow with context length. For the original GLM-5, a single history requires approximately 2.74 GiB of main-attention KV state at 32K tokens and 10.97 GiB at 128K tokens, assuming BF16 storage. Retaining many long histories requires either reducing the memory each history consumes or extending cache capacity beyond GPU memory.

Reduce the memory needed for each history

The cache footprint depends on both the model’s attention architecture and the precision used to store its state. Multi-head latent attention (MLA) stores a compressed representation of attention state; the GLM-5 example above already benefits from this approach. KV cache quantization can reduce memory further by using fewer bits per stored value, where supported.

Figure 6 illustrates how a smaller footprint creates more room for reuse. A 64-GiB budget reserved for KV state holds five histories at 11 GiB each. If each history’s footprint is halved to 5.5 GiB, the same budget fits eleven. The histories retain the same context length, so the additional capacity comes from storing their state more compactly.

Keeping more histories available increases opportunities for prefix reuse. The number of agents that can actively generate responses also depends on compute, memory bandwidth, and response-time requirements.

Retain KV state beyond GPU memory

Hierarchical caching extends cache capacity by retaining KV state in host DRAM and disk, as introduced in panel 3 of Figure 1. Histories can remain available between requests while GPU memory is freed for active inference. Figure 7 shows how this state moves. The model executor reads and updates KV in GPU memory, while a cache controller coordinates transfers between tiers. State that is not immediately needed can be offloaded to host DRAM and disk. When a later request reuses that state, a host-cache hit can restore it directly to the GPU. The illustrated disk path first brings it into host DRAM, then transfers it to GPU memory.

The controller can prefetch state and overlap transfers with other work, but each attention operation needs its required KV entries ready in GPU memory. Restoration therefore introduces a trade-off: retaining more history avoids repeated computation, while retrieving it adds transfer work and potential delay. Cache placement should preserve useful state while meeting the application’s response-time requirements.

Route requests to workers that can reuse KV state

Across a pool of serving workers, each worker may hold different KV state and have a different queue. The same request can therefore face different waiting times and require different amounts of prefill work depending on where it runs. Cache-aware routing considers both when choosing a destination.

To make that choice, the router needs to know which prefixes each worker can reuse. Panel 4 of Figure 1 shows how KV-cache events provide this information. Inference engines publish KV events when KV blocks are stored or removed. In the router, a KV indexer uses these updates to track cached prefixes, and a worker selector combines the resulting prefix matches with worker load information.

Figure 8 illustrates how these two inputs affect the decision. Worker 1 has a short 0.1-second queue but no matching prefix, so it needs 3.0 seconds of prefill, for a combined 3.1 seconds. Worker 2 has a longer 0.4-second queue, but its matching prefix reduces prefill to 0.1 seconds, for a total of 0.5 seconds. Choosing Worker 2 reduces queue-plus-prefill time by 2.6 seconds because the saved prefill work outweighs the additional wait.

Turn KV reuse into lower agent costs

An agent’s growing history represents work the inference system has already performed. KV caching preserves the attention state produced by that work so it can support later tokens and requests. As an agent takes more steps, keeping this state reusable becomes increasingly valuable.

Sustaining that reuse requires coordination across the serving system. Prefix caching carries state across requests, hierarchical caching extends how much of it can be retained, and cache-aware routing connects new requests to workers that can reuse it. Together, these mechanisms determine how much earlier computation remains useful as contexts grow and more agents run concurrently. Keeping useful KV state available where and when it is needed helps turn that accumulated work into lower inference costs.

Put KV reuse to work for your agents with FriendliAI. Talk to our engineers about reducing inference costs as your contexts and workloads grow.