Rethinking AI Inference Kubernetes Cluster Consistency with Atomic State Reconciliation

In late October 2025, AWS experienced a major service disruption that rippled across EC2, DynamoDB, load balancers, and multiple dependent services. The root cause was not a hardware failure or a capacity shortage, but a latent race condition in an automated control system that left critical DNS state inconsistent and unrecoverable without manual intervention.

What made the incident notable was its nature: an inconsistent control-plane state that automation could neither safely detect nor reconcile. Even after the issue was identified, recovery took hours as dependent systems struggled to converge back to a consistent view of the world.

Incidents like this highlight a fundamental reality of large-scale distributed systems: reliability failures often stem from how state is managed. In practice, modern AI infrastructure is almost universally operated on top of Kubernetes (K8S) clusters, which serve as the de facto plane for managing compute, memory, and GPUs. While Kubernetes is highly resilient, it is not immune to control-plane state inconsistencies.

When such inconsistencies occur in AI inference clusters where inference services demand low latency, high availability, and predictable behavior, the impact can be severe, leading to incorrect resource allocation, stalled workloads, and user-visible reliability failures.

At FriendliAI, we observed these issues firsthand while operating large-scale AI infrastructure. This led us to deeply analyze how reconciliation inconsistencies arise in Kubernetes-based systems and to design a principled solution. The result is Garen, a system built on Atomic State Reconciliation (ASR), which enforces atomicity during reconciliation to prevent clusters from entering inconsistent states. This work will be presented at EuroSys 2026.

Reliability as a First-Class Requirement for AI Inference

Introducing Garen AI inference services operate under strict constraints. Latency budgets are tight, traffic patterns are often bursty, and resource utilization, especially for GPUs, must be both efficient and predictable. In this environment, control-plane correctness directly affects service quality.

Clusters continuously reconcile desired state with actual state. When this process is correct, the system adapts smoothly to load changes and failures. When it is not, even brief inconsistencies can cascade into widespread service degradation.

For AI inference systems, reliability is therefore not an afterthought or a purely operational concern. It is a fundamental correctness property.

Why Reconciliation Fails in Practice

Most cluster controllers follow a simple control loop: observe current state, compute desired changes, and apply updates. This loop is executed concurrently across controllers and resources, often under partial failures and retries.

Figure 1 illustrates this mechanism through a Kubernetes deployment scaling workflow. When the desired replica count changes in response to fluctuating traffic, the deployment controller detects the divergence between the current and target states and sends requests to the API server to create or remove pods. This action initiates a cascade of asynchronous updates handled by multiple independent controllers, including the scheduler and the kubelet. Operating in parallel, these controllers each reconcile specific fields of the pod state, collectively driving the cluster toward eventual convergence with the desired configuration.

In Kubernetes, reconciliation can break down under concurrency and failure. Controllers may:

• Read stale state
• Apply partial updates
• Interleave with other controllers operating on overlapping resources

These behaviors open the door to race conditions and inconsistent intermediate states. Once such states are observed, controllers may make incorrect decisions that further destabilize the cluster.

Figure 2(b) demonstrates a concrete failure scenario involving the Coscheduler. In this case, the `minMember` (the minimum number of pods required for a distributed job) is updated to match the workload requirements. However, the scheduler fails to capture this change in time and continues to operate on outdated configuration. As a result, it attempts to schedule the job based on incorrect requirements, ultimately leading to a failure.

Real-World Consistency Bugs

The FriendliAI team analyzed 51 real-world consistency bugs observed in Kubernetes controllers, spanning both native controllers and widely used custom controllers. These bugs are not hypothetical corner cases; they represent failures that occurred in real production environments.

Many of these inconsistencies resulted in severe consequences, including service outages, resource leaks, data loss, and controller malfunctions (for a detailed quantitative analysis, please refer to our EuroSys 2026 paper). A significant portion of the bugs involved multi-object reconciliation under concurrent updates, a pattern that is increasingly common in modern, highly dynamic clusters.

Equally concerning is their resolution status. Many of the identified bugs remain unfixed today. Some proposed fixes were rejected due to backward compatibility concerns, while others imposed unacceptable performance overhead. This underscores a deeper limitation: existing reconciliation mechanisms lack the fundamental guarantees needed to address these issues cleanly.

Takeaway: Even “battle-tested” Kubernetes clusters suffer from systemic reconciliation flaws, and many of these issues cannot be fully resolved through controller-specific patches or operational workarounds alone.

The Impact on AI Inference Clusters

AI inference clusters amplify the cost of reconciliation failures. Resource allocation errors can lead to:

• GPUs being double-allocated or stranded
• Inference jobs stalling, restarting, or being evicted
• Latency spikes and degraded tail performance during traffic surges

Our analysis at FriendliAI showed that many of these failures stem from subtle control-plane inconsistencies that existing reconciliation mechanisms fail to prevent.

Atomic State Reconciliation (ASR)

Atomic State Reconciliation addresses this problem by ensuring that each reconciliation step appears atomic. A controller transition either applies entirely or not at all, even in the presence of concurrency and failures.

By eliminating unsafe intermediate states, ASR prevents entire classes of consistency bugs that are otherwise difficult to detect and debug. This shifts reconciliation from a best-effort mechanism to a strongly consistent system primitive.

How Garen Achieves ASR

Controller Decomposition

Garen analyzes controller logic and decomposes it into small conditional reconciliation units. Each unit explicitly captures the minimal state required for a given reconciliation step.

This decomposition allows Garen to reason precisely about which parts of the cluster state are involved in each operation.

Minimal-State Conflict Detection

Rather than relying on coarse-grained locking, Garen tracks conflicts only within the minimal state footprint of each reconciliation. This enables strong consistency without sacrificing scalability or throughput.

Dry-Run Validation

Before committing any update, Garen uses Kubernetes’ dry-run API to validate state transitions. This ensures that all constraints are satisfied and that only safe, consistent updates are applied to the live cluster.

Evaluation Results

Garen demonstrates that achieving strong consistency is practical in real-world systems:

• 18 previously unresolved Kubernetes consistency bugs were fixed, without manual controller changes
• The system scales across cluster sizes and API request rates
• End-to-end overhead remains below 3%

These results show that strong consistency guarantees can be introduced without compromising performance, an essential requirement for production AI infrastructure. For a detailed description of the experimental setup, workloads, and evaluation results, please refer to the EuroSys 2026 paper.

Figure 4 demonstrates the efficiency of minimal-state (i.e., fine-grained) conflict detection. In contrast to baseline approaches that rely on naïve, coarse-grained detection and suffer severe latency spikes due to repeated retries from spurious conflicts, Garen incurs negligible overhead (<1%) relative to native Kubernetes. Even at 20× trace speeds, Garen remains stable, whereas the naïve ASR baseline experiences a latency explosion of up to 1202%.

Furthermore, Figure 5 confirms that the minimal-state conflict detection is essential for cluster scalability. Typically, as cluster size grows, the overlapping concurrent updates become more frequent, causing baseline approaches to incur a high rate of spurious conflicts. Garen, however, effectively eliminates these unnecessary conflicts by restricting conflict detection to the minimal relevant state footprint. As a result, it maintains less than 3% overhead across all evaluated cluster sizes, demonstrating that minimal-state conflict detection successfully reconciles strong consistency with high scalability.

Implications for AI Infrastructure

Garen addresses reconciliation consistency as a fundamental systems design problem rather than a controller-specific implementation detail. For AI inference infrastructure, this enables:

• More predictable and self-healing clusters
• Stronger alignment between infrastructure behavior and inference SLAs
• Reduced operational burden caused by rare but catastrophic failure modes
• This approach aligns closely with FriendliAI’s goal of building reliable, scalable, and predictable AI inference systems.

Looking Forward

As AI workloads grow more latency-sensitive and operationally complex, reconciliation can no longer be best-effort. Atomicity must be treated as a first-class design principle in cluster management.

Garen and Atomic State Reconciliation represent a step toward that future. At FriendliAI, we will continue investing in systems research and engineering to ensure that AI infrastructure remains reliable—not just under normal conditions, but under the concurrency and failure modes that define real production environments.

References

[1] AWS, “Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region,” 2025. [Online]. Available: https://aws.amazon.com/message/101925/. Accessed: Dec. 22, 2025.

[2] M. Kim, A. Shin, J. Maeng, M. Jeon, and B.-G. Chun, “Garen: Reliable Cluster Management with Atomic State Reconciliation,” in Proceedings of the 21st European Conference on Computer Systems (EuroSys ’26), Edinburgh, UK, Apr. 27–30, 2026. New York, NY, USA: ACM, 2026, pp. 1–16. doi: 10.1145/3767295.3769383.