GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
gittensor-model-hub
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
gittensor-model-hub
Model Tree
gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090
Input Modalities
Output Modalities
Supported Functionality
At concurrency 1 this model is weight-bandwidth bound, not compute bound. Measured on the parent: 18.80 GiB of resident weights at 81.6 tok/s implies a 1.65 TB/s read rate against the RTX 5090's 1.79 TB/s spec — about 92% of peak. Decode is streaming the whole model once per token.
lm_head is a full-vocabulary (248,320 × 5,120) GEMM evaluated on every token, so it is 2.54 GB of that per-token read. Quantizing it to NVFP4 cuts it to 0.72 GB.
The bandwidth model predicted 89.7 tok/s; measured 88.45 — within 1.4%. On this hardware, bytes removed from the weight read path convert almost linearly into tokens per second.
Note the distinction: embeddings are the same 2.54 GB but are a gather (~10 KB/token), so quantizing them would save capacity, not decode speed. lm_head is the only large BF16 block on the per-token critical path.
Same 20 items per task, seed 20260815, thinking on, temperature=1.0, top_p=0.95, 24k generation cap — both arms run through the same harness on SGLang.
| Task | Parent | This variant |
|---|---|---|
| GPQA Diamond | 10/20 | 13/20 |
| AIME 2025 | 11/20 | 12/20 |
| MMLU-Pro | 17/20 | 17/20 |
| Overall | 38/60 (63%) | 42/60 (70%) |
Read this as "no degradation", not "an improvement." At n=20 per task with temperature=1.0, a 4-item difference is well inside noise, and 6–9 items per task hit the 24k truncation cap on both arms. The claim supported by this data is that 4-bit logits did not measurably hurt quality — not that they helped.
These numbers are also not comparable to the parent card's published smoke, which was measured on vLLM 0.27.1; this harness runs on SGLang. Both arms here share one harness, so the comparison between them is valid.
The v2 DSpark drafter was trained against the parent's logits, so changing lm_head could have degraded acceptance. It did not — acceptance is flat to slightly up:
| Domain | Parent | This variant |
|---|---|---|
| Math | 4.388 | 4.496 |
| Coding | 3.804 | 3.864 |
| Long-context | 2.334 | 2.474 |
| Chat | 2.450 | 2.363 |
| Held-out overall | 2.886 | 2.904 |
| Agentic tool calling |
Acceptance measured per request with the prefix cache flushed, from cumulative verify counters.
Identical to the parent — no flag changes:
bash
sglang serve --model-path gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4 \--trust-remote-code --tp-size 1 \--context-length 65536 --kv-cache-dtype fp8_e4m3 \--attention-backend flashinfer --chunked-prefill-size 2048 \--mamba-radix-cache-strategy extra_buffer_lazy --mamba-ssm-dtype bfloat16 \--mem-fraction-static 0.90 --max-running-requests 2 \--speculative-algorithm DSPARK \--speculative-draft-model-path gittensor-model-hub/Qwen3.8-27B-DSpark-NVFP4 \--speculative-dspark-block-size 7 --speculative-draft-model-quantization modelopt_fp4 \--reasoning-parser qwen3 --tool-call-parser qwen3_coder
The native MTP head is retained, so vLLM speculation still works.
lm_head.weight (BF16, 248320 × 5120) replaced by four NVFP4 tensors:
markdown
lm_head.weight U8 (248320, 2560) packed E2M1lm_head.weight_scale F8_E4M3 (248320, 320) per-group-16 scaleslm_head.weight_scale_2 F32 () global weight scalelm_head.input_scale F32 () activation scale, amax 49.75
The activation scale comes from measuring real lm_head inputs on the served parent (amax 49.75, p50 40.0 — a clean distribution, max/p50 = 1.24, no outlier tail). ModelOpt 0.45 performed the quantization so the E2M1 packing and UE4M3 scale layout match what the runtime expects.
lm_head was also removed from both exclude lists — hf_quant_config.json → quantization.exclude_modules and config.json → quantization_config.ignore. Missing the second one causes a silent fallback to an unquantized layer, surfacing as Parameter lm_head.input_scale not found in params_dict and then a 5120-vs-2560 shape error.
Apache 2.0, same as the parent and the Qwen3.8-27B base model.
| 3.891 |