GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
PhilipEriksson
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
PhilipEriksson
Model Tree
PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled
Input Modalities
Output Modalities
Supported Functionality
| File | Size |
|---|---|
model_backbone.safetensors | 892 MB total |
model_mtp.safetensors | (multi-token-prediction head) |
recipe.yaml | quantization recipe |
config.json, tokenizer, chat template |
Format is compressed-tensors — auto-detected by vLLM, no --quantization flag needed.
vLLM (Blackwell / SM120 for the FP4 tensor-core path):
bash
vllm serve PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled-NVFP4 \ --max-model-len 16384transformers:
python
from transformers import AutoModelForImageTextToText, AutoTokenizer model = AutoModelForImageTextToText.from_pretrained( "PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled-NVFP4", dtype="bfloat16", device_map="cuda",)tok = AutoTokenizer.from_pretrained( "PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled-NVFP4")Loading requires compressed-tensors>=0.15.0. Without it you get
ImportError: compressed-tensors>=0.15.0 is required for compressed-tensors quantization
— the artifact is fine, the environment is missing the loader.
Creating an NVFP4 quant and serving one are different toolchains. Creation is llm-compressor and runs on any CUDA GPU; FP4 tensor cores only matter for running W4A4 inference, which needs vLLM with SM120 kernels compiled in.
Three prompts (arithmetic, code, logic) run through every artifact of this model, same
sampler settings, max_new_tokens=700:
| Artifact | Size | Expected answer found |
|---|---|---|
| BF16 (baseline) | 1.7 GB | 3/3 |
| GGUF Q8_0 | 834 MB | 3/3 |
| GGUF Q4_K_M | 542 MB | 3/3 |
| EXL3 4.0 bpw | 1.08 GB | 3/3 |
| EXL3 3.0 bpw | 1.02 GB | 2/3 |
| NVFP4 | 892 MB | 3/3 |
NVFP4 matches the BF16 baseline on this set. Three prompts is a smoke test, not a benchmark.
llm-compressor oneshot with QuantizationModifier(targets="Linear", scheme="NVFP4"),
calibrated on 256 samples of the corpus the model was trained on — not generic web text,
because the activation ranges that matter are the ones the model actually sees.
Ignore list (per architecture, Qwen3_5ForConditionalGeneration):
yaml
- "re:.*linear_attn.conv1d" # small conv — FP4 dispatch costs more than it saves- "re:.*linear_attn.in_proj_ba" # small proj — same- "lm_head"The older NVIDIA ModelOpt modelopt_fp4 format is not used — it caused
k_scale/v_scale KeyErrors and garbled MoE output.
apache-2.0 license.Collection: Qwen3.5-0.8B-Opus-Reasoning-Distilled
Base model: PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled
LoRA adapter: PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled-LoRA
GGUF: PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled-GGUF
EXL3: PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled-EXL3
vllm serve PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled-NVFP4 \ --max-model-len 16384from transformers import AutoModelForImageTextToText, AutoTokenizer model = AutoModelForImageTextToText.from_pretrained( "PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled-NVFP4", dtype="bfloat16", device_map="cuda",)tok = AutoTokenizer.from_pretrained( "PhilipEriksson/Qwen3.5-0.8B-Opus-Reasoning-Distilled-NVFP4")- "re:.*linear_attn.conv1d" # small conv — FP4 dispatch costs more than it saves- "re:.*linear_attn.in_proj_ba" # small proj — same- "lm_head"