Same Model, Different Results: Why Your Inference Provider Matters

Choosing an inference provider matters as much as choosing the model itself. When you use a closed model through its direct API, model capability and inference come as one package. With open-weight models, you can choose among providers that serve the same model or deploy it on your own infrastructure.

Providers use different software, hardware, and operating practices to serve the same model. These differences affect answer quality, wait times, and the cost of completing a task. They also shape how the service handles failures and processes your data. This article explains the engineering behind these differences and uses the five dimensions in Figure 1 to guide your evaluation of an inference partner.

Quality: Preserve the model’s capability

The same model can deliver different answer quality and tool-use reliability across providers. Differences come from how each provider configures the model, executes computations, and parses output. Figure 2 highlights three engineering choices that affect how much of the model’s capability reaches your application.

Quantization: Preserve accuracy at lower precision

Quantization represents numerical values with fewer bits, reducing memory requirements and potentially accelerating inference. But providers serving the same “4-bit” model differ in quantization methods, affecting how much accuracy the model preserves. Figure 3 shows the differences in delivered quality across providers serving four open-weight models. Artificial Analysis’s Endpoint Accuracy Index compares each endpoint against a self-hosted reference across tool calling, reasoning, and long-context evaluations. A score of 100 matches that model’s reference performance. Some endpoints remain close to this baseline, while others fall more than 25 index points below it

GPU kernels: Ensure correct execution

GPU kernels are the programs that execute a model’s calculations, including matrix multiplication and attention. Providers select and optimize these implementations for their models and hardware. How a kernel combines operations or accumulates intermediate results affects numerical behavior. Implementation bugs or excessive numerical error can alter the model’s output even when its weights and advertised precision remain unchanged.

A June 2026 University of Washington study examined 90 real-world silent errors reported in vLLM and SGLang. Their causes spanned framework code, model implementations, kernel backends, and numerical precision. In one developer-confirmed SGLang case, the researchers traced outputs that differed from the reference implementation to a FlashInfer kernel. Such failures can degrade answers while the service continues returning responses without an error.

Preserving quality therefore requires validating the computations behind each optimization. Providers should compare optimized implementations against trusted references and verify model accuracy across supported hardware, batch sizes, and context lengths. Repeating these checks after serving-system changes helps catch regressions before they reach applications.

Tool-call handling: Parse reliably across models

Models generate tool calls in different formats. The serving system must parse the function name and arguments, then translate them into the schema the application expects. It must also format previous calls and tool results correctly when preparing the next request. Errors in either direction can disrupt an agent’s workflow, leaving the application without a usable action even if the request succeeded.

An October 2025 vLLM investigation into Kimi K2 illustrates the problem. Prompt-formatting mismatches interfered with generation, while the parser discarded calls whose identifiers used an unexpected format. In the report’s final test, 99.925% of requests completed successfully, but only 76% of triggered tool calls passed schema validation. In March 2026, OpenRouter reported that routing requests toward providers with stronger measured quality reduced tool-call error rates by 88% for GLM-5 and 80% for GLM-4.7.

Constrained decoding can limit a model to supported tools and enforce each tool’s argument schema. Providers should test the full round trip for each model, including streamed responses and parallel calls. The parser must deliver each call intact to the application, and the model must receive the matching result on the next turn.

Quality assurance requires validating every serving change

Preserving model quality requires ongoing validation. Changes to precision, kernels, or tool handling should pass regression checks against a trusted baseline before deployment. Representative application tasks should confirm that the updated system continues to produce accurate answers and usable actions. These checks make quality a release requirement as the serving stack evolves.

Performance: Deliver faster responses and higher throughput

A provider’s performance determines both how long users wait and how much work the service can complete. Even with the same model, differences in hardware, caching, execution, and scheduling affect response latency, generation speed, and aggregate throughput. For agents, inference delays accumulate across model calls, while available throughput determines how many workflows the service can sustain. Figure 4 shows how provider engineering improves these dimensions.

Time to first token: Reduce the wait before generation

Before returning its first token, the serving system must route a request to available capacity and process its input. Busy queues and large, uncached prompts increase this wait. On Vercel AI Gateway, FriendliAI’s hourly median TTFT averaged 266 ms from October 13 to 20, 2026, the lowest among 22 providers serving GLM-5.3. See our Vercel AI Gateway blog for a detailed comparison among inference providers.

Prefix caching reduces input-processing time by reusing the model state computed for a matching prompt prefix, such as shared instructions or conversation history. Cache-aware routing directs requests to workers that already hold the relevant state. Routing must also account for worker load, since a long queue can outweigh the time saved by a cache hit. Balancing cache reuse with available capacity helps lower TTFT by reducing both repeated computation and time spent waiting for service.

Output speed: Complete responses faster

Output speed determines how quickly a response or tool call finishes generating. It can be expressed as time per output token (TPOT), the average time between successive output tokens. Lower TPOT means faster generation, with larger time savings for longer outputs.

Output speed depends on GPU hardware, memory bandwidth, and how efficiently the serving engine executes the model. FriendliAI accelerates execution with custom GPU kernels and draft-model speculative decoding. Figure 5 illustrates how output speed varies across providers serving the same model. In the figure, FriendliAI records a median output rate of 132 tokens/s, the highest among the eight endpoints shown and 36% higher than the next endpoint at 97 tokens/s.

Throughput: Serve more requests at the required speed

Throughput measures the total output token rate across concurrent requests. Higher throughput lets a provider support more users and agent workflows, as long as individual requests still meet response-time targets. Scheduling and batching determine how efficiently those requests share GPUs. Continuous batching, pioneered by FriendliAI, lets new requests join the active batch as soon as others finish, improving utilization.

In FriendliAI’s GLM-5.3-Flash benchmark, Friendli Inference delivered approximately 2.1x vLLM’s aggregate throughput on four NVIDIA B200 GPUs while both engines maintained an output speed of 100 tokens/s per request.

Cost: Compare total spend per successful task

There are two common ways to pay for inference: model API usage and dedicated capacity. Model APIs typically charge for input and output tokens, with discounts for eligible cached input. With dedicated capacity, you pay for GPU resources allocated to your workload, including time when those resources are idle. Figure 6 illustrates these two purchasing models.

API usage: Pay less for repeated input

Even at identical token prices, providers can produce different bills because they serve different shares of input from cache. A higher cache hit rate means more input receives the discounted rate, increasing savings on repeated instructions, conversation history, and repository context.

A larger key-value (KV) cache can retain more previously computed context before it must evict older entries. This gives subsequent calls more opportunity to reuse matching prefixes, especially as agent histories and concurrent usage grow. Effective cache management and routing help turn that capacity into cache hits and lower input charges.

Dedicated capacity: Get more from GPU time

Higher sustained throughput produces more output from the same paid GPU time while meeting the application’s output-speed target. At the same GPU-hour price, doubling throughput on the same hardware halves GPU cost per output token.

Smart autoscaling matches paid capacity to demand. It adds replicas as traffic grows and removes unneeded capacity during quiet periods. Policies that account for startup time and keep enough replicas ready can protect response times during bursts without paying for peak capacity continuously.

Reliability: Deliver through spikes and failures

A provider’s reliability determines how consistently requests complete within your application’s time limits. Traffic spikes, worker failures, and infrastructure changes all test that ability. For agents, an interrupted call can stall an entire workflow. Figure 7 shows how queueing and routing help protect request completion.

Spare capacity and bounded queues help absorb sudden demand while autoscaling brings additional workers online. Queue limits and timeouts should reflect the application’s response-time requirements, with clear rejection behavior when the service cannot accept more work.

Routing must track which workers are ready to serve requests. Health checks identify unhealthy workers so new or retried requests can move to available capacity, while retry limits and backoff help prevent recovery attempts from amplifying an overload. This coordination also matters during deployments and scaling. New workers should receive traffic only after they are ready, and workers being removed should stop accepting new requests while finishing those already in progress.

Data control: Define how and where data is handled

The same model can be served under different rules for data location, access, retention, and training use. These rules come from the provider’s infrastructure, policies, and agreements, so provider choice affects which workloads a business can run and under what conditions.

Retention policies define what data a provider stores and for how long, while training policies govern whether inputs and outputs can be used to train models. A provider can prohibit training while retaining content for diagnostics or abuse monitoring. Zero data retention (ZDR) addresses whether request content is retained after processing, subject to the policy’s scope and stated exceptions.

Deployment choices shape where processing occurs and who can access the system. Hosted APIs operate on the provider’s infrastructure under its access controls. Deploying in a customer’s own cloud can place processing within chosen regions and network boundaries, but a managed deployment may still grant the provider permissions for maintenance or support.

The value of an inference partner

The model’s capabilities reach users through the system serving it. Provider decisions affect the resulting quality, wait times, how much work fits within the budget, and whether the service meets reliability and data requirements. The value of those decisions becomes visible in the application’s actual workload.

That value comes from engineering across the serving system. Precision, kernels, caching, scheduling, routing, and deployment interact, so an improvement in one area can change the trade-offs elsewhere. A capable inference partner understands those relationships and translates them into better product outcomes.

The work continues after launch. New models introduce different architectures and behavior, while changing demand exposes new bottlenecks. The provider’s ability to diagnose problems, adapt its serving stack, and validate changes becomes part of the product’s ability to improve.

See what FriendliAI can deliver for your workload. Talk to a FriendliAI engineer about your model, traffic, and production requirements.