GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
LostGentoo
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
LostGentoo
Model Tree
Qwen/Qwen3.5-4B
Input Modalities
Output Modalities
Supported Functionality
This release exists to provide a known base + auditable calibration for Qwen3.5-4B in ModelOpt NVFP4 form. It publishes the exact preset, public calibration mix, seed, hashes, and measured speed/quality so the recipe can be re-run independently.
| Item | Value |
|---|---|
| Base | Qwen/Qwen3.5-4B |
| Tool | NVIDIA ModelOpt 0.45.0 |
| Preset | NVFP4_DEFAULT_CFG (max calib; group size 16) |
| KV cache quant | none (kv_cache_quant_algo: null) |
| Excludes | lm_head, vision / multimodal modules, linear_attn.conv1d / in_proj_* (ModelOpt defaults + export) |
| Calibration | 512 public samples, seed 20260730 |
| Mix | ~70% HuggingFaceH4/ultrachat_200k train_sft + ~30% HuggingFaceFW/fineweb-edu sample-10BT, chat-templated, max seq 4096 |
| Calib SHA-256 | 762d54c48ad951318dc641aeb0f2091729bb6f8008ef89126c74ac41eea591b2 (calib.jsonl) |
| Quant host | RTX 5060 Ti (SM120), CUDA 12.8, driver 610.43, torch 2.11.0+cu128 |
Scripts live with the producer workspace (build_calib.py, quantize_modelopt.py). AWQ_LITE was attempted but export hit a Qwen fuse/pre_quant_scale shape error; DEFAULT (max) is the published path.
bash
python -m vllm.entrypoints.openai.api_server \--model LostGentoo/Qwen3.5-4B-NVFP4 \--served-model-name LostGentoo/Qwen3.5-4B-NVFP4 \--max-model-len 8192 \--gpu-memory-utilization 0.90 \--trust-remote-code
vLLM auto-detects ModelOpt NVFP4 (quantization=modelopt_fp4). On Blackwell you should see:
text
Using CutlassNvFp4LinearKernel for NVFP4 GEMM
Hardware: native FP4 GEMM requires NVIDIA Blackwell. Non-Blackwell will not get the intended speedup.
Tip: Qwen3.5 may emit a “Thinking Process” preamble. Prefer higher
max_tokensfor evals, or disable thinking in the chat template if your stack supportsenable_thinking=false//no_think.
Server-side deltas via vllm_probe.py delta + ctxsweep / prefill_probe (unique prompts).
| Conc | Agg tok/s |
|---|---|
| 1 | 87 |
| 4 | 218 |
| 8 | 292 |
| 16 | 341 |
| 24 | 355 |
Whole-matrix server delta decode ≈ 334 tok/s. Prefill peak @6144 ≈ 17.1k tok/s (conc1 fit ≈ 16.2k). 0 preemptions / sweep cell errors. Kernel: CutlassNvFp4LinearKernel.
| Check | Result |
|---|---|
| 50 synthetic product-support spot prompts | 50/50 coherent replies |
IFEval (chat_smoke, n=541, thinking on) | prompt_level_strict 38.3%, inst_level_strict 49.5% |
IFEval (chat_smoke, n=541, enable_thinking=false) | prompt_level_strict 72.1%, inst_level_strict 79.9% |
| Needle-in-haystack | Pass @4096 with long completion budget; thinking-on was the main source of flakiness |
MMLU-Pro (max_samples=1000, enable_thinking=false) |
Service qwen35-4b-axion-nvfp4-b6000, vLLM v0.24.0 image + stdlib proxy, max-num-seqs=128. Native fp4_gemm.
| Decode @6144 | Agg tok/s |
|---|---|
| C1 | 162 |
| C16 | 1006 |
| C64 | 1486 |
Prefill peak @6144: 69.7k tok/s. 0 errors / preemptions. Scaled to minScale=0 after the spot.
5060 Ti remains a consumer SM120 signal; B6000 is the G4-class capacity number.
model.safetensors — NVFP4 weights (+ unquantized excluded modules)hf_quant_config.json — ModelOpt producer metadataconfig.json — architectures: [Qwen3_5ForConditionalGeneration], quantization_config.quant_algo: NVFP4ForCausalLM checkpoint (vLLM text-only path still works).Qwen/Qwen3.5-4B)hf_quant_config.json layoutbibtex
@misc{lostgentoo_qwen35_4b_nvfp4,title = {Qwen3.5-4B NVFP4 (ModelOpt, public calib)},author = {LostGentoo},year = {2026},url = {https://huggingface.co/LostGentoo/Qwen3.5-4B-NVFP4}}
| NVFP4 31.2%, bf16 28.7%, AWQ 28.5% |
MMLU-Pro (max_samples=3000, enable_thinking=false) | NVFP4 31.27%, bf16 29.23% (Δ +2.03 pp) |
| MMLU-Pro gate vs bf16 | Fails the plan threshold: NVFP4 stays ~+2 pp vs bf16 on both 1k and 3k subsets (target <=1.5 pp) |