GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
axteam
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
axteam
Model Tree
Nanbeige/Nanbeige4.3-33B
Input Modalities
Output Modalities
Supported Functionality
nvfp4-pack-quantized.chat_template identical to the original.Fidelity to the BF16 original on an internal mixed EN/KO set (100 prompts).
| Metric | This model | BF16 original |
|---|---|---|
| Perplexity (lower = closer) | 1.3594 | 1.2232 (+11.1%) |
| Top-1 next-token agreement | 90.2% | 99.1% |
⚠️ Note: noticeable 4-bit degradation; prefer 8-bit (FP8 / AWQ INT8).
| Runtime | vLLM — the architecture is in vLLM; this build not run here yet. |
| vLLM install | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| GPU | Full FP4 (W4A4) speed on Blackwell — B200/GB200 (sm100), B300 (sm103), RTX 50xx / RTX PRO 6000 (sm120). Turing–Hopper (sm75–90) fall back to weight-only FP4 (slower, vLLM warns). V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| Memory | Weights: 22.1 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU. |
| Transformers (without vLLM) | transformers>=4.56.0 + compressed-tensors>=0.15.0 (same Llama architecture as MiniCPM5; not run here for Nanbeige). |
| trust_remote_code | not needed |
Start an OpenAI-compatible server:
bash
vllm serve axteam/Nanbeige4.3-33B-NVFP4 --max-model-len 32768Query it:
bash
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "axteam/Nanbeige4.3-33B-NVFP4", "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}], "max_tokens": 512}'Offline inference in Python:
python
from vllm import LLM, SamplingParams llm = LLM(model="axteam/Nanbeige4.3-33B-NVFP4", max_model_len=32768)params = SamplingParams(max_tokens=1024)out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)print(out[0].outputs[0].text)Thinking mode is on by default in the chat template; to turn it off pass
chat_template_kwargs={"enable_thinking": False}(Pythonllm.chat(..., chat_template_kwargs=...), or"chat_template_kwargs": {"enable_thinking": false}in the request body).
The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:
bash
vllm serve axteam/Nanbeige4.3-33B-NVFP4 --kv-cache-dtype fp8Quantized derivative of Nanbeige/Nanbeige4.3-33B (apache-2.0).
Nanbeige/Nanbeige4.3-33B 를 NVFP4 (W4A4, FP4) 로 양자화한 서빙용(4-bit) 모델입니다. 작성: axteam.
nvfp4-pack-quantized.vllm serve axteam/Nanbeige4.3-33B-NVFP4 --max-model-len 32768 로 OpenAI 호환 서버(포트 8000), 또는 Python LLM(model="axteam/Nanbeige4.3-33B-NVFP4").chat(...) (위 예시). thinking 끄기: chat_template_kwargs={"enable_thinking": False}.--kv-cache-dtype fp8.curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "axteam/Nanbeige4.3-33B-NVFP4", "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}], "max_tokens": 512}'from vllm import LLM, SamplingParams llm = LLM(model="axteam/Nanbeige4.3-33B-NVFP4", max_model_len=32768)params = SamplingParams(max_tokens=1024)out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)print(out[0].outputs[0].text)