How Modern LLM Architectures Reshape Agent Inference

Agent workloads involve long, growing contexts across multiple model calls. As a task progresses, successive calls share a common prefix, with new messages and tool results appended to the end. Supporting these workloads requires both model architectures and inference runtimes that scale efficiently as context grows.

Modern LLM architectures address this scaling challenge by changing how they represent and access history, such as reducing per-token state or compressing history into a fixed-size recurrent state. These approaches change what an inference runtime must retain and reuse. The latest open-weight models combine multiple approaches, making state management a concern that extends beyond the conventional KV cache.

In this article, we examine the architectural and runtime changes needed to scale inference for long-context agent workloads. Using Kimi K3 as a case study, we show how hybrid architectures represent a shared prefix as a bundle of attention and recurrent states. We then explore the implications for prefix reuse, cache management, and speculative decoding, focusing on how runtimes must handle these states to realize the model architecture’s efficiency gains. The central consequence: for hybrid models, a matching prefix no longer guarantees reusable state, and in an agent loop the tokens most at risk of recomputation are the model’s own previous response.

Growing Contexts and Caching Prefixes

OpenRouter’s State of AI 2025 report analyzed more than 100 trillion tokens and found that average prompt length rose from roughly 1.5K tokens in early 2024 to more than 5K in late 2025. Programming workloads had particularly long inputs: code-related requests routinely exceeded 20K input tokens.

In agent workloads, much of the input repeats across model calls. Manus reports an average input-to-output token ratio of around 100:1 for its agent and calls KV-cache hit rate the single most important metric for a production agent. Prefix caching exploits this repetition by preserving the state computed for an earlier request. In a conventional full-attention model, this state primarily consists of the keys and values stored for past tokens in the KV cache. When the state for a shared prefix remains available, the runtime reuses it and prefills only the newly added suffix.

This reuse avoids recomputing the prefix, but it does not eliminate the costs of increasing context. In full-attention models, KV cache memory still grows with context length, and both subsequent prefill and decode steps must attend to an expanding history. Scaling inference to longer contexts therefore requires more than reusing prior computation: it also depends on how the model represents history and which past positions it accesses. The next section examines architectural choices that reduce these remaining costs.

Three Architectural Approaches to Long-Context Scaling

Store Less State per Token

One approach reduces the state cached for each token while retaining access to every past token position. Standard multi-head attention (MHA) caches separate keys and values for each attention head. Multi-query attention (MQA) shares a single key head and value head across all query heads, while grouped-query attention (GQA) shares them within groups of query heads. Both reduce the cache footprint and the amount of KV data read during decoding.

Multi-head latent attention (MLA) reduces per-token state through low-rank compression. Introduced in DeepSeek-V2, MLA carries through DeepSeek-V3 and R1 and has been adopted by other labs, including Moonshot AI’s Kimi K2 and Z.ai’s GLM-5. It caches a joint latent representation of content keys and values, together with a separate key carrying rotary position embeddings (RoPE). Weight absorption folds the expansion matrices into the query and output projections, so decoding can attend directly over the compressed latents. Prefill may temporarily expand keys and values to use more compute-efficient attention kernels.

Read Fewer Past Tokens

A second approach reduces the number of past positions each query attends to. Sliding-window attention restricts each query to a fixed-size window of recent tokens, bounding attention computation and KV reads per query as context grows. Mistral 7B applied it in every layer, while recent models such as Gemma 3 and OpenAI’s gpt-oss interleave sliding-window layers with full-attention layers. Once a token falls outside the window, that layer no longer needs its KV state to continue the sequence, so the runtime can release it from the active cache. Prefix reuse introduces a separate requirement: a later request can resume from a shared prefix only if each sliding-window layer still has the KV state for the window immediately preceding the reuse point. Evicting that state can therefore force recomputation even when the prefix tokens match.

Dynamic sparse attention selects tokens or blocks from a broader history, allowing successive queries to attend to different, potentially distant positions. DeepSeek Sparse Attention (DSA), introduced in DeepSeek-V3.2, uses a lightweight indexer to pick the tokens each query attends to; GLM-5 and DeepSeek-V4 build on it. A token skipped by one query may be needed by the next, so the runtime may still need to retain the full KV history even though each query attends to only a subset. Fewer attention reads therefore do not, by themselves, imply a proportionally smaller cache. The runtime must also select positions and gather noncontiguous KV entries, and that selection has its own cost.

Compress History into a Fixed-Size State

A third approach represents history as a fixed-size recurrent state. Linear-attention mechanisms such as Kimi Delta Attention (KDA), along with state space model (SSM) layers, can process tokens this way during decoding. Each new token updates the existing state, combining new information with what has already been retained, rather than adding a separate token-level KV entry. For a given layer configuration, the state’s size remains independent of context length. Production models use these layers in hybrids with full attention: Qwen3-Next and Qwen3.5 interleave Gated DeltaNet layers with full attention at roughly 3:1, NVIDIA’s Nemotron 3 pairs Mamba-2 layers with attention, and Kimi Linear and Kimi K3 use KDA.

Updating it also does not automatically preserve earlier versions, so the runtime cannot recover an earlier state simply by deleting a suffix of cache entries. Recovery requires explicit support, such as restoring a checkpoint and replaying subsequent tokens as needed. Checkpoints also allow later requests to reuse shared prefixes at saved boundaries. The active recurrent state therefore stays fixed in size, while retaining checkpoints for reuse and recovery consumes additional memory.

Hybrid State and Prefix Reuse in Kimi K3

Hybrid architectures combine mechanisms that retain history in different forms. Kimi K3’s main repeating pattern places three KDA layers before one full-attention Gated MLA layer. Both process the same token sequence, but they leave the runtime with different state to manage.

Attention and Recurrent Layers

Gated MLA adds an output gate to MLA while preserving full attention over the token history. Each Gated MLA layer caches a compressed KV representation for every processed token, so its cache grows with context length. These layers preserve direct access to individual past positions.

Each KDA layer maintains a matrix-valued recurrent state and short convolution buffers that retain recent projected activations to mix information from neighboring tokens. Both update as tokens arrive, but their sizes remain fixed. Using KDA in most layers therefore reduces the amount of cache that grows with context length. Across the model, the MLA caches, recurrent states, and convolution buffers form the state bundle needed to continue from a processed prefix.

Aligning State for Prefix Reuse

Reusing a prefix requires all components of this bundle to correspond to a common token boundary. To resume after a particular token, the runtime needs each Gated MLA layer’s KV entries through that position and each KDA layer’s recurrent and convolution states after processing it. The components need not share a physical layout, but they must describe the same processed prefix. A longer MLA cache cannot substitute for missing KDA state.

A long token match therefore does not guarantee equally long reuse. If the matching MLA cache extends beyond the latest usable KDA checkpoint, the runtime must fall back to an earlier complete bundle. Cache management must consequently track which prefix boundaries have a complete bundle available for reuse, as the next section examines.

Managing State Bundles in the Inference Runtime

Hybrid architectures require the runtime to coordinate state with different sizes, update patterns, and lifetimes. Alongside efficient kernels, serving these models requires decisions about where computation can resume, which state to retain for future requests, and how to recover after speculative verification.

Prefix Reuse: The Recomputation Gap

Prefix lookup must distinguish the end of the matching token sequence from the latest boundary with a complete reusable state bundle. If the cache holds only KV and state checkpoints, the runtime restores the latest complete bundle and prefills all subsequent tokens through the model, including both the remaining matching tokens and the new suffix. The matched tokens between the two boundaries form the recomputation gap: work repeated because the state needed to resume farther ahead was not retained. Retaining each KDA layer’s intervening inputs would let the runtime replay only the recurrent updates instead, at the cost of storing those inputs.

Cache Management: Checkpoint Retention and Placement

A hybrid cache manager must accommodate token-level KV blocks, recurrent snapshots, and convolution buffers. A prefix checkpoint associates recurrent and convolution snapshots at a boundary with the KV blocks covering that prefix. Those KV blocks can be shared across checkpoints, so retaining another boundary need not duplicate the entire KV prefix.

Checkpoint frequency trades memory and state writes against future recomputation. A KDA recurrent snapshot can be much larger than one token’s MLA cache entry, making frequent snapshots expensive. Retaining selected boundaries, such as prompt ends or frequently shared prefixes, concentrates storage on states likely to be reused. For an agent call, this choice determines how much of the previous response must be processed again. If the next request repeats the prior prompt and response before adding a tool result, a checkpoint available only at the prior prompt boundary leaves the response in the recomputation gap. The runtime already holds a complete bundle when it finishes generating, so persisting it there costs a single snapshot and can shrink the gap to the few tokens the chat template re-encodes at the turn boundary.

Placement determines the cost of restoring retained state. GPU memory avoids transfers but leaves less capacity for active requests; host memory or storage frees GPU capacity at the cost of retrieval. Evicting a required component can remove a reuse boundary even when other components remain cached, so the manager must track availability across the complete bundle. In distributed serving, routing must also weigh the location of reusable state against server load and transfer cost.

Speculative Decoding: Recovery at the Accepted Boundary

Speculative decoding lets a drafter propose several candidate tokens for the target model to verify in one pass. Once verification determines the accepted prefix, the runtime must exclude the rejected suffix from future computation. For token-level attention, it can shorten the valid KV range. Recurrent state updated through rejected tokens needs an additional recovery mechanism, and convolution buffers must also return to the accepted boundary.

One option saves recurrent and convolution snapshots at each candidate boundary, allowing the runtime to select the accepted state. This avoids replay but increases storage and state writes. Another keeps a starting checkpoint and compact inputs for each candidate, then replays only the accepted updates. Kimi K3’s reported implementation uses cached projected inputs to rebuild accepted-token states on-chip. These temporary records support recovery during decoding; they are separate from checkpoints retained for reuse across requests.

ReplaySSM makes the checkpoint and buffered inputs the state representation itself. The checkpoint together with the accepted portion of the buffer represents the accepted state, so rollback can adjust a buffer pointer without immediately reconstructing a full state. Periodic checkpoint updates reduce state writes during both ordinary and speculative decoding. ReplaySSM provides up to a 2.14× speedup over vLLM’s speculative-decoding baseline on hybrid models such as Nemotron 3 and Qwen3.5. Evaluating speculative decoding in hybrid models therefore requires accounting for state storage, memory traffic, and recovery computation alongside drafting cost, verification cost, and token acceptance rate.

From Architectural Efficiency to Serving Performance

Efficient long-context serving requires model architectures and inference runtimes to scale together. Architectures reduce the cost of retaining and accessing history; runtimes must preserve those gains across prefix reuse, cache management, and speculative decoding. For hybrid models, this means keeping KV, recurrent state, and convolution state consistent at every reuse and recovery boundary.

For agents, these improvements matter only when they reduce task completion time and cost. At FriendliAI, we serve models on our own inference stack, built to understand these architectural changes at the system level and optimize for them end to end. As new architectures arrive, our stack evolves with them, so agent workloads get the efficiency each model was designed to deliver.

👉 Deploy your agent workload on Friendli Suite