GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
stoyani
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
stoyani
Model Tree
primitive-ai/Nanbeige4.3-33B-FP8
Input Modalities
Output Modalities
Supported Functionality
kv_cache_scales.safetensors contains 156 float32 scalars: Q, K and V scales
for each of the 52 attention layers, named
model.layers.{0..51}.self_attn.attn.{q,k,v}_scale.
Ranges were observed at the attention input, after positional rotation of
Q/K, using the exact FP8 weights with BF16 KV cache. Each scale is
1.10 * observed_absolute_maximum / 448, rounded upward to float32.
Calibration used 14 samples, with the longest containing 68,190 tokens.
Calibration inputs and per-sample artifacts are not distributed.
The included Q scales do not imply that BF16 queries should be rescaled.
Validated with vLLM 0.32.0 plus the FlashInfer query-dtype correction on an RTX PRO 6000 Blackwell. Unpatched 0.32.0 native FlashInfer can incorrectly apply non-unit Q scales to BF16 queries and produce incorrect output. Use a runtime containing that correction; see vLLM issue #63043 and the proposed fix #53754. This checkpoint does not patch vLLM. Other runtime/backend combinations have not been validated for this release.
After satisfying that runtime requirement, an example for the tested 96 GB GPU is:
bash
VLLM_USE_RUST_FRONTEND=0 VLLM_USE_V2_MODEL_RUNNER=1 VLLM_USE_FLASHINFER_SAMPLER=0 \vllm serve stoyani/Nanbeige4.3-33B-FP8-KVcal \ --dtype bfloat16 \ --kv-cache-dtype fp8 \ --max-model-len 524288 \ --gpu-memory-utilization 0.95 \ --enable-chunked-prefill \ --enable-prefix-caching \ --prefix-match-unit 16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choice \ --default-chat-template-kwargs '{"enable_thinking": true, "preserve_thinking": true, "tool_call_format": "xml"}' \ --override-generation-config '{"temperature": 1.0, "top_p": 0.95}'Follow the original model card
for sampling and chat-template options. It explicitly recommends the qwen3
reasoning parser for Nanbeige's thinking format; the parser name does not
select the model. The example sets agent/tool defaults; for reasoning/chat,
the original card recommends temperature 0.6 and top_p 0.95. These scales
target per-tensor FP8 KV, not NVFP4 weights or other cache quantization schemes.
All 156 scales matched runtime readback. Short arithmetic, repetition, structured JSON, tool-call and thinking checks passed. Synthetic retrieval also passed with approximately 200K and 520K input tokens, within the model's 524,288-token context.
These are serving and retrieval checks, not evidence of task-quality equivalence to BF16 KV. No matched BF16-KV quality study was performed. Calibration coverage and synthetic retrieval do not guarantee accuracy for arbitrary workloads or every long-context task.
The tokenizer and generation settings are unchanged from the pinned Primitive
AI release. config.json enables static FP8 KV scaling; the index adds the
156 scale tensors and their 624 bytes of tensor data. SHA256SUMS lists the
release files, excluding itself.
Apache-2.0. This is a modified distribution of the Primitive AI FP8 conversion of Nanbeige4.3-33B. Credit to Nanbeige LLM Lab for the original model and Primitive AI for the FP8 weight quantization.