GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
abhishek085
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
abhishek085
Model Tree
Qwen/Qwen3-4B
Input Modalities
Output Modalities
Supported Functionality
Batch size 1, idle GPU, sequential single-decision latency (scripts/quant/endpoint_latency.py, the 60-row diagnostic set):
| serving stack | precision | p50 | p95 | decisions/s |
|---|---|---|---|---|
| Hugging Face Transformers (in-process) | bf16 | 65.9 ms | - | 15.1 |
| vLLM | bf16 | 61.6 ms | 69.9 ms | 16.2 |
| vLLM | NVFP4, MLP only (this repo) | 37.1 ms | 38.8 ms | 27.1 |
1.66x faster than the same checkpoint in bf16 on the same hardware, comparing like-for-like through vLLM. File size drops from 7.6 GB (bf16) to 4.0 GB.
The only accuracy check run on this quantized variant so far is JevBench's public tiers (231 items); it has not yet been scored on the own-splits/external-set battery the bf16 card reports.
| tier | bf16 | NVFP4 (this repo) | Δ |
|---|---|---|---|
| easy (48) | 1.000 | 1.000 | 0.0 |
| standard (72) | 0.847 | 0.847 | 0.0 |
| hard (111) | 0.523 | 0.477 | -4.6 |
Easy and standard tiers are held exactly; the hard tier gives up 4.6 points. This is a much smaller cost than the same MLP-only recipe had on the 1.7B checkpoint (9-13 points across external sets there, not shipped as a quantized release) — the 4B has more redundancy to spare.
Caveat on the raw logits: the calibration temperature saved in calibration.json here is copied unchanged from the bf16 checkpoint.
Quantization noise shifts the raw logit distribution somewhat (seen more severely on the 1.7B, where a full-precision-style temperature under-
calibrated the quantized model); if you rely on the model's confidence for thresholding, refit the temperature on your own calibration data
against this specific quantized checkpoint rather than trusting the bundled one as-is.
bash
docker run -d --gpus all --ipc host -p 8000:8000 \ -v /path/to/this/checkpoint:/model \ vllm/vllm-openai:nightly-aarch64 /model --served-model-name spark-s1-4b-v5-nvfp4 \ --gpu-memory-utilization 0.5 --max-model-len 4096The open-spark-Jev gateway can front this endpoint directly:
python -m open_spark_jev.serve.gateway --backend openai --upstream http://127.0.0.1:8000/v1 --model /path/to/this/checkpoint, which also
applies calibration.json and speaks the same POST /v1/decide contract as the bf16 model.
bash
docker run --rm --gpus all -v $PWD:/work -w /work nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13 \ python3 scripts/quant/ptq_nvfp4.py --model /work/checkpoints/v5-4b --tokenizer /work/models/Qwen3-4B \ --calib /work/runs/quant/calib_prompts_v5.jsonl --out /work/engines/spark-s1-4b-v5-MLP_ONLY_CFG --cfg NVFP4_MLP_ONLY_CFGFull source: github.com/abhishek085/open-spark-jev.
See abhishek085/spark-s1-4b-v5 for the base model's limitations (Jev-style decisions
only, RLCD not applied, etc.), all of which carry over here. In addition: not yet accuracy-checked outside JevBench's public tiers; the bundled
calibration temperature is inherited from bf16, not refit for this checkpoint's own logit distribution.
Apache-2.0 (the Qwen/Qwen3-4B backbone is Apache-2.0).
docker run -d --gpus all --ipc host -p 8000:8000 \ -v /path/to/this/checkpoint:/model \ vllm/vllm-openai:nightly-aarch64 /model --served-model-name spark-s1-4b-v5-nvfp4 \ --gpu-memory-utilization 0.5 --max-model-len 4096docker run --rm --gpus all -v $PWD:/work -w /work nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc13 \ python3 scripts/quant/ptq_nvfp4.py --model /work/checkpoints/v5-4b --tokenizer /work/models/Qwen3-4B \ --calib /work/runs/quant/calib_prompts_v5.jsonl --out /work/engines/spark-s1-4b-v5-MLP_ONLY_CFG --cfg NVFP4_MLP_ONLY_CFG