GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
Ttimms
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Ttimms
Model Tree
Kwaipilot/KAT-Coder-V2.5-Dev
Input Modalities
Output Modalities
Supported Functionality
Weight-only quantization (NVFP4A16) dequantizes on the fly and computes in
bf16 — there is no hardware path for a weight-only scheme to reach a GPU's
native FP4×FP4 tensor-core instructions, on any GPU. Quantizing activations
too (this build) does reach those native kernels
(FlashInferCutlassNvFp4LinearKernel for dense layers, FLASHINFER_CUTLASS
for MoE experts, confirmed via vLLM's own kernel-selection logging). The
question this build answers: does native FP4×FP4 compute actually beat a
dequant-then-bf16-compute fallback on consumer Blackwell (SM120, e.g. RTX
5070 Ti/5080/5090) for a real agentic-coding decode workload?
5 interleaved process invocations per arm (warmup discarded), median + range reported, under both isolated eager-mode execution (cleanly isolates kernel dispatch, but understates real throughput by ~7x on this hardware) and the same PIECEWISE CUDA-graph configuration the NVFP4A16 model serves with in production:
| HumanEval | HumanEval+ | MBPP+ | decode, eager | decode, PIECEWISE (production) | |
|---|---|---|---|---|---|
| NVFP4A16 (sibling release) | 95.7% | 90.9% | 89.9% | 18.8 tok/s | 142.5 tok/s |
| W4A4 (this build) | 92.07% | 89.02% | 91.01% | 14.5 tok/s | 119.2 tok/s |
On this hardware, for this single-stream (batch=1) workload: NVFP4A16 is faster (0.84x for W4A4 under the production PIECEWISE configuration, 0.77x in isolated eager mode). Both numbers are fast in absolute terms — 119 tok/s is roughly 17-20x typical human reading speed and comfortably interactive for coding use; the gap is relative to the sibling build, not a usability threshold. Accuracy is mixed rather than uniformly different: W4A4 is lower on HumanEval and HumanEval+, and very slightly higher on MBPP+ (344/378 vs 340/378 — one problem's difference on a 378-problem set). For reference, QSpec-era literature (INT4-generation quantization) reported W4A4 losing up to 38.73% on HumanEval; that collapse did not reproduce here, consistent with NVFP4's per-16-block scaling and FP8 scale factors being a better-conditioned format than INT4-era quantization.
Same checkpoint size either way (12.4532 GiB here vs 12.4512 GiB for A16).
We only measured single-stream (batch=1) decode, matching our own agentic use case. We have not measured batch sizes above 1, where the native kernel's throughput/latency characteristics may differ from the dequant path's — if you test that, we'd like to hear what you find.
Two findings surfaced while measuring this, independent of the headline numbers:
gpu_memory_utilization, the NVFP4A16/Marlin arm needed a
higher utilization setting to reliably allocate KV cache than this W4A4
build did at the same checkpoint size — Marlin's dequantize-on-the-fly
path appears to need more non-weight runtime workspace than the native
FP4 kernel path does.Loads and serves like the NVFP4A16 sibling (same architecture,
transformers/vllm requirements, SM120/compute-capability-12.0
requirement, no CPU offload needed at 12.45 GiB). See the sibling release's
model card for full serving instructions, environment requirements, and the
REAP-pruning background — this card documents only what differs about this
build.
python
from vllm import LLM, SamplingParamsllm = LLM(model="Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4-W4A4",dtype="bfloat16",language_model_only=True,trust_remote_code=True,)
Apache 2.0, inherited from the base model Kwaipilot/KAT-Coder-V2.5-Dev and
its upstream lineage.