GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
axteam
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
axteam
Model Tree
Qwen/Qwen3.5-2B
Input Modalities
Output Modalities
Supported Functionality
nvfp4-pack-quantized.model.visual.*), the multi-token-prediction head (mtp.*), all RMSNorms, token embeddings, the LM head, the entire linear-attention (GatedDeltaNet/SSM) path — FP4 is too coarse for the SSM weights, so the full linear_attn.* stays BF16. Left unquantized to preserve quality (about 55% of all parameters in this build).chat_template / generation_config are identical to the original. config.json is the original plus a quantization_config block.This is post-training quantization (PTQ) — no retraining. AWQ uses a small calibration set (no gradient) to protect salient weight channels before rounding.
Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.
| Benchmark | BF16 (original) | FP8-dynamic | AWQ-8bit | NVFP4A16 | NVFP4 (this repo) | AWQ-4bit |
|---|---|---|---|---|---|---|
| MMLU (5-shot) | 53.33 | 48.18 (90.3%) | 53.54 (100.4%) | 54.27 (101.8%) | 55.22 (103.5%) | 60.38 (113.2%) |
| GSM8K (5-shot, strict) | 58.23 | 56.63 (97.3%) | 56.56 (97.1%) | 54.51 (93.6%) | 51.71 (88.8%) | 54.06 (92.8%) |
| ARC-Challenge (0-shot) | 41.47 | 41.98 (101.2%) | 41.30 (99.6%) | 41.04 (99.0%) | 39.51 (95.3%) | 39.59 (95.5%) |
| Winogrande (5-shot) | 64.56 | 63.22 (97.9%) | 64.48 (99.9%) | 64.48 (99.9%) | 61.09 (94.6%) | 62.51 (96.8%) |
| KMMLU (5-shot, direct) | 43.29 | 42.58 (98.4%) | 43.20 (99.8%) | 42.01 (97.1%) | 41.14 (95.0%) | 42.10 (97.3%) |
| HAE-RAE (0-shot) | 45.65 | 45.00 (98.6%) | 45.10 (98.8%) | 44.91 (98.4%) | 42.07 (92.2%) | 41.06 (90.0%) |
| Average | 51.09 | 49.60 (97.1%) | 50.70 (99.2%) | 50.21 (98.3%) | 48.46 (94.8%) | 49.95 (97.8%) |
⚠️ Note: average recovery is 94.8% — noticeable degradation at this precision; prefer an 8-bit build if quality matters most.
Fidelity to the BF16 original on an internal mixed English/Korean set (100 prompts, BF16 greedy answers scored under each build — same harness for every row).
| Metric | This model | BF16 original |
|---|---|---|
| Perplexity (lower = closer) | 2.0399 | 1.8515 (+10.2%) |
| Top-1 next-token agreement | 85.8% | 98.9% |
These measure fidelity to the base model, not absolute task scores.
⚠️ Note: this small dense model is sensitive to 4-bit quantization (+10.2% perplexity here). For production we recommend the 8-bit builds (FP8 or AWQ INT8) of this model.
| Runtime | vLLM — the architecture is in vLLM ≥ 0.17.0; tested here with vLLM 0.32.0. |
| vLLM install | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| GPU | Full FP4 (W4A4) speed on Blackwell — B200/GB200 (sm100), B300 (sm103), RTX 50xx / RTX PRO 6000 (sm120). Turing–Hopper (sm75–90) fall back to weight-only FP4 (slower, vLLM warns). V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| Memory | Weights: 3.1 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU. |
| Transformers (without vLLM) | transformers>=5.2.0 + compressed-tensors>=0.15.0 (checked on CPU with Qwen3.5-0.8B FP8 / AWQ-4bit / NVFP4A16 / NVFP4). |
| trust_remote_code | not needed |
Start an OpenAI-compatible server:
bash
vllm serve axteam/Qwen3.5-2B-NVFP4 --max-model-len 32768Query it:
bash
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "axteam/Qwen3.5-2B-NVFP4", "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}], "max_tokens": 512}'Offline inference in Python:
python
from vllm import LLM, SamplingParams llm = LLM(model="axteam/Qwen3.5-2B-NVFP4", max_model_len=32768)params = SamplingParams(max_tokens=1024)out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)print(out[0].outputs[0].text) # image input (URL or data: URI)messages = [{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/your-image.png"}}, {"type": "text", "text": "Describe this image."},]}]out = llm.chat(messages, params)print(out[0].outputs[0].text)Thinking mode is off by default in this model's chat template; to turn it on pass
chat_template_kwargs={"enable_thinking": True}(Pythonllm.chat(..., chat_template_kwargs=...), or"chat_template_kwargs": {"enable_thinking": true}in the request body). Text-only serving: add--limit-mm-per-prompt '{"image": 0, "video": 0}'to skip loading the vision encoder's memory budget.
The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:
bash
vllm serve axteam/Qwen3.5-2B-NVFP4 --kv-cache-dtype fp8Quantized derivative of Qwen/Qwen3.5-2B (Apache-2.0). All original weights, tokenizer and chat template belong to the Qwen team.
Qwen/Qwen3.5-2B의 NVFP4 (W4A4) 양자화 빌드 · 제작 axteam
| 항목 | 내용 |
|---|---|
| 원본 모델 | Qwen/Qwen3.5-2B |
| 양자화 방식 | NVFP4 (W4A4, 가중치·활성화 FP4) |
| 저장 형식 | compressed-tensors nvfp4-pack-quantized |
| 모델 구조 | 하이브리드 어텐션(선형 어텐션 + 풀 어텐션) · Dense · 멀티모달(비전) · MTP 헤드 |
| 용량 | 약 3.1 GB |
| 보정 데이터 | 샘플 256개 (한국어 위주 혼합 코퍼스) |
model.visual.*), MTP 헤드(mtp.*), 모든 RMSNorm, 토큰 임베딩, LM 헤드, 선형 어텐션(GatedDeltaNet/SSM) 경로 전체(linear_attn.*, FP4로는 정밀도 부족) — 전체 파라미터의 약 55%config.json은 원본에 quantization_config 블록만 추가--tensor-parallel-size N으로 분산vllm serve axteam/Qwen3.5-2B-NVFP4 --max-model-len 32768 — OpenAI 호환 APILLM(model="axteam/Qwen3.5-2B-NVFP4", max_model_len=32768).chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고chat_template_kwargs={"enable_thinking": True} 전달(Python은 llm.chat() 인자, HTTP는 요청 본문의 "chat_template_kwargs": {"enable_thinking": true})image_url 항목 사용--limit-mm-per-prompt '{"image": 0, "video": 0}' — 비전 타워용 메모리 예약을 생략해 메모리 절약vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "axteam/Qwen3.5-2B-NVFP4", "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}], "max_tokens": 512}'from vllm import LLM, SamplingParams llm = LLM(model="axteam/Qwen3.5-2B-NVFP4", max_model_len=32768)params = SamplingParams(max_tokens=1024)out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)print(out[0].outputs[0].text) # image input (URL or data: URI)messages = [{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/your-image.png"}}, {"type": "text", "text": "Describe this image."},]}]out = llm.chat(messages, params)print(out[0].outputs[0].text)