GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
axteam
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
axteam
Model Tree
Qwen/Qwen3.5-27B
Input Modalities
Output Modalities
Supported Functionality
nvfp4-pack-quantized.model.visual.*), the multi-token-prediction head (mtp.*), all RMSNorms, token embeddings, the LM head, the entire linear-attention (GatedDeltaNet/SSM) path — FP4 is too coarse for the SSM weights, so the full linear_attn.* stays BF16. Left unquantized to preserve quality (about 32% of all parameters in this build).chat_template / generation_config are identical to the original. config.json is the original plus a quantization_config block.This is post-training quantization (PTQ) — no retraining. AWQ uses a small calibration set (no gradient) to protect salient weight channels before rounding.
Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.
| Benchmark | BF16 (original) | FP8-dynamic | AWQ-8bit | NVFP4A16 (this repo) | NVFP4 | AWQ-4bit |
|---|---|---|---|---|---|---|
| MMLU (5-shot) | 86.02 | 86.01 (100.0%) | 86.05 (100.0%) | 86.07 (100.1%) | 86.01 (100.0%) | 86.08 (100.1%) |
| GSM8K (5-shot, strict) | 37.00 | 38.06 (102.9%) | 34.80 (94.1%) | 81.43 (220.1%) | 71.27 (192.6%) | 82.87 (224.0%) |
| ARC-Challenge (0-shot) | 61.18 | 61.69 (100.8%) | 61.60 (100.7%) | 60.84 (99.4%) | 60.58 (99.0%) | 60.84 (99.4%) |
| Winogrande (5-shot) | 80.43 | 80.03 (99.5%) | 80.58 (100.2%) | 79.64 (99.0%) | 79.48 (98.8%) | 80.11 (99.6%) |
| KMMLU (5-shot, direct) | 69.79 | 69.83 (100.1%) | 69.84 (100.1%) | 69.61 (99.7%) | 69.19 (99.1%) | 68.98 (98.8%) |
| HAE-RAE (0-shot) | 76.99 | 77.36 (100.5%) | 77.64 (100.8%) | 75.89 (98.6%) | 75.71 (98.3%) | 76.44 (99.3%) |
| Average | 68.57 | 68.83 (100.4%) | 68.42 (99.8%) | 75.58 (110.2%) | 73.71 (107.5%) | 75.89 (110.7%) |
| Average without GSM8K | 74.88 | 74.98 (100.1%) | 75.14 (100.4%) | 74.41 (99.4%) | 74.19 (99.1%) | 74.49 (99.5%) |
Note on GSM8K: this task is scored on free-form generation, so the score depends on whether the model answers in exactly the format the scorer expects. For this model it swings widely between builds of the same model (GSM8K 34.8–82.9), and some quantized builds score far above the BF16 original — that reflects answer-format adherence, not quantization quality. The second average leaves it out.
Fidelity to the BF16 original on an internal mixed English/Korean set (100 prompts, BF16 greedy answers scored under each build — same harness for every row).
| Metric | This model | BF16 original |
|---|---|---|
| Perplexity (lower = closer) | 1.3779 | 1.3603 (+1.3%) |
| Top-1 next-token agreement | 95.6% | 99.3% |
These measure fidelity to the base model, not absolute task scores.
| Runtime | vLLM — the architecture is in vLLM ≥ 0.17.0; tested here with vLLM 0.32.0. |
| vLLM install | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| GPU | NVIDIA Turing (sm75) or newer — weight-only FP4, no FP4 tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| Memory | Weights: 28.6 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU. |
| Transformers (without vLLM) | transformers>=5.2.0 + compressed-tensors>=0.15.0 (checked on CPU with Qwen3.5-0.8B FP8 / AWQ-4bit / NVFP4A16 / NVFP4). |
| trust_remote_code | not needed |
Start an OpenAI-compatible server:
bash
vllm serve axteam/Qwen3.5-27B-NVFP4A16 --max-model-len 32768Query it:
bash
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "axteam/Qwen3.5-27B-NVFP4A16", "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}], "temperature": 0.6, "top_p": 0.95, "top_k": 20, "max_tokens": 512}'Offline inference in Python:
python
from vllm import LLM, SamplingParams llm = LLM(model="axteam/Qwen3.5-27B-NVFP4A16", max_model_len=32768)params = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, max_tokens=1024)out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)print(out[0].outputs[0].text) # image input (URL or data: URI)messages = [{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/your-image.png"}}, {"type": "text", "text": "Describe this image."},]}]out = llm.chat(messages, params)print(out[0].outputs[0].text)Recommended sampling (from the original
generation_config.json):temperature=0.6,top_p=0.95,top_k=20. Thinking mode is on by default in the chat template; to turn it off passchat_template_kwargs={"enable_thinking": False}(Pythonllm.chat(..., chat_template_kwargs=...), or"chat_template_kwargs": {"enable_thinking": false}in the request body). Text-only serving: add--limit-mm-per-prompt '{"image": 0, "video": 0}'to skip loading the vision encoder's memory budget.
The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:
bash
vllm serve axteam/Qwen3.5-27B-NVFP4A16 --kv-cache-dtype fp8Quantized derivative of Qwen/Qwen3.5-27B (Apache-2.0). All original weights, tokenizer and chat template belong to the Qwen team.
Qwen/Qwen3.5-27B의 NVFP4A16 (W4A16) 양자화 빌드 · 제작 axteam
| 항목 | 내용 |
|---|---|
| 원본 모델 | Qwen/Qwen3.5-27B |
| 양자화 방식 | NVFP4A16 (W4A16, FP4 가중치) |
| 저장 형식 | compressed-tensors nvfp4-pack-quantized |
| 모델 구조 | 하이브리드 어텐션(선형 어텐션 + 풀 어텐션) · Dense · 멀티모달(비전) · MTP 헤드 |
| 용량 | 약 28.6 GB |
| 보정 데이터 | 샘플 256개 (한국어 위주 혼합 코퍼스) |
model.visual.*), MTP 헤드(mtp.*), 모든 RMSNorm, 토큰 임베딩, LM 헤드, 선형 어텐션(GatedDeltaNet/SSM) 경로 전체(linear_attn.*, FP4로는 정밀도 부족) — 전체 파라미터의 약 32%config.json은 원본에 quantization_config 블록만 추가--tensor-parallel-size N으로 분산vllm serve axteam/Qwen3.5-27B-NVFP4A16 --max-model-len 32768 — OpenAI 호환 APILLM(model="axteam/Qwen3.5-27B-NVFP4A16", max_model_len=32768).chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고temperature=0.6, top_p=0.95, top_k=20 (원본 generation_config.json 기준)chat_template_kwargs={"enable_thinking": False} 전달(Python은 llm.chat() 인자, HTTP는 요청 본문의 "chat_template_kwargs": {"enable_thinking": false})image_url 항목 사용--limit-mm-per-prompt '{"image": 0, "video": 0}' — 비전 타워용 메모리 예약을 생략해 메모리 절약vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "axteam/Qwen3.5-27B-NVFP4A16", "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}], "temperature": 0.6, "top_p": 0.95, "top_k": 20, "max_tokens": 512}'from vllm import LLM, SamplingParams llm = LLM(model="axteam/Qwen3.5-27B-NVFP4A16", max_model_len=32768)params = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, max_tokens=1024)out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)print(out[0].outputs[0].text) # image input (URL or data: URI)messages = [{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/your-image.png"}}, {"type": "text", "text": "Describe this image."},]}]out = llm.chat(messages, params)print(out[0].outputs[0].text)