GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
sokada4
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
sokada4
Model Tree
deepseek-ai/DeepSeek-V4-Flash-0731
Input Modalities
Output Modalities
Supported Functionality
| Method | GPTQ (GPTQModel 7.3.4) |
| Bits | 4 |
| Group size | 128 |
| Symmetric | yes |
desc_act | false |
lm_head quantized | no |
| Pack dtype | int32 |
128 samples (32 each of zh / en / ja / code), seq_len=2048:
allenai/c4 — zh, en, ja splits (mangled/mis-decoded documents filtered out)codeparrot/codeparrot-clean-valid — Python sourcezh/en/ja/code were weighted evenly. seq_len=2048 was chosen above this model's
sparse-indexer index_topk (512) so the top-k pruning path is actually exercised
during calibration, while staying within the indexer's dense-attention memory budget
on the quantization rig.
This model requires a vLLM build with DeepSeek-V4 GPTQ support on the Ampere
(SM86, e.g. RTX 3090/4090-class, non-Hopper) backend, which is not yet present in
either upstream vLLM or the unmodified vllm-dsv4-ampere community fork. The
required patch is available at:
https://github.com/okdshin/vllm-dsv4-ampere/tree/gptq-w4a16-support
Apply the main-port/vllm overlay from that branch on top of a vanilla vLLM
checkout, then serve as usual, e.g.:
bash
vllm serve sokada4/DeepSeek-V4-Flash-0731-GPTQ-Int4-V1 \--trust-remote-code \--kv-cache-dtype fp8 \--tensor-parallel-size 4 \--enable-expert-parallel \--enforce-eager \--max-num-seqs 8 \--max-model-len 4096
(--cpu-offload-gb may be needed depending on available VRAM; the full checkpoint
is ~153GB.)
Measured on a 4x GPU Ampere rig (--cpu-offload-gb 20, vllm bench serve),
compared against the unquantized fp8 checkpoint:
| concurrency | metric | fp8 (unquantized) | GPTQ 4bit | delta |
|---|---|---|---|---|
| 1 | TTFT mean | 2345.56 ms | 1184.35 ms | -49% |
| 1 | TPOT mean | 335.33 ms | 317.98 ms | -5% |
| 1 | output tok/s | 2.73 | 3.02 | +11% |
At concurrency=1, decode is bound by PCIe transfer of CPU-offloaded weights rather than GPU compute/bandwidth, so the TPOT improvement from quantization is modest; at concurrency=8, batching amortizes that fixed cost and the quantization benefit is much larger.
MIT, inherited from the base model.
| 8 |
| TTFT mean |
| 8434.68 ms |
| 4486.18 ms |
| -47% |
| 8 | TPOT mean | 735.13 ms | 432.72 ms | -41% |
| 8 | output tok/s | 9.34 | 16.11 | +72% |