GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
simaai
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
simaai
Model Tree
Qwen/Qwen3-VL-2B-Instruct
Input Modalities
Output Modalities
Supported Functionality
| Component | Method | Weight format | Targets |
|---|---|---|---|
| Decoder Linear layers | GPTQ | Symmetric INT4, group size 256, static act-order | 196 |
lm_head | RTN | Symmetric INT8, per output channel | 1 |
| Vision encoder and merger Linear layers | RTN | Symmetric INT8, per output channel | 104 |
| Other parameters, including input embeddings | Unchanged | Source BF16 | — |
The INT8 head is quantized directly from the original BF16 output weights, rather than from an INT4 reconstruction. It has one scale per vocabulary output row: [151936, 1]. The head remains separate from the BF16 input embeddings. Activations are not prequantized.
Decoder calibration uses HuggingFaceH4/ultrachat_200k, revision 8049631c405ae6576f93f445c6b8166f76f5505a, train_sft[:4096]: the original chat template, deterministic order, 512 full packed spans of 1,024 tokens, batch size 1, seed 42, no shuffle. GPTQ uses block size 128, dampening fraction 0.01, and static act-order. RTN vision and head quantization do not require calibration data. Exact targets and the calibration token hash are in recipe.yaml.
Matched evaluation on 18 October 2026 used Transformers 5.10.1, BF16 GPU arithmetic with decompressed quantized weights, SDPA, the original processor/tokenizer, and no repetition penalty. All four Slurm array tasks completed successfully. Aggregate results and settings are in evaluation_results.json.
| Checkpoint | Full WikiText-2 word perplexity ↓ | MMStar accuracy ↑ | Text repetition flags ↓ |
|---|---|---|---|
| Original BF16 | 16.5131 | 37.33% | 2/106 |
| UltraChat GPTQ with INT4 head (comparison) | 21.2537 | 33.00% | 8/106 |
| This checkpoint: GPTQ with RTN INT8 head | 18.6151 | 35.67% | 5/106 |
Perplexity uses the full wikitext lm-eval task, EleutherAI/wikitext_document_level, wikitext-2-raw-v1, without an example limit; maximum sequence length 2,048, batch size 1. Relative to BF16, this checkpoint adds 2.1021 word perplexity (+12.73%). The INT8 head improves matched perplexity over the INT4-head comparison (21.2537 → 18.6151).
MMStar uses all 1,500 examples, images resized to 448 × 448, greedy generation, at most 128 new tokens, and VLMEvalKit exact_matching, with no external judge. VLMEvalKit revision: 7055d3010c38ccb5dcae1bc9535ca19c7fe5d79f. The change from BF16 is −1.67 percentage points. Many responses hit the output limit: BF16 514/1,500; this checkpoint 582/1,500. These are budget-limited scores and are not directly comparable with earlier higher-resolution evaluations.
The conversational suite includes three 20-turn conversations, fresh writing prompts, record-update/JSON checks, six short reasoning questions, and three image follow-ups: 109 responses per model, of which 106 are text-only. Repetition flags are heuristics and were reviewed in the transcripts; token-limit hits alone are not repetition. This checkpoint still loops on some prompts, passes 0/6 record-update/JSON checks, and gives correct final answers on 4/6 short reasoning questions. Those small checks do not establish broad reasoning accuracy. Image/text generation and complete MMStar inference passed; these are GPU results, not Modalix runtime validation.
Download the original BF16 model at the pinned revision, then run the included standalone script in a compatible CUDA environment:
bash
hf download Qwen/Qwen3-VL-2B-Instruct \ --revision 89644892e4d85e24eaac8bacfd4f463576704203 \ --local-dir Qwen3-VL-2B-Instruct python quantize.py \ --model-path Qwen3-VL-2B-Instruct \ --output-dir Qwen3-VL-2B-Instruct-GPTQ-Safetensorsquantize.py combines the original two stages: reproduce the GPTQ decoder/vision checkpoint, then replace its temporary INT4 head with RTN INT8 from the supplied BF16 model. No private model-cache path or separately downloaded helper script is required. Newly reproduced artifacts require their own evaluation.
The tokenizer assets are preserved byte-for-byte from the original BF16 checkpoint. Verified tokenizer.json SHA-256:
text
a5d85b6dcc535e6b93115a9ef287e6132fdbf30270da6218194ba742261173c7text
Python: 3.13.2CUDA: 12.8torch: 2.11.0+cu128transformers: 5.10.1llmcompressor: 0.12.0compressed-tensors: 0.17.1datasets: 5.0.0See versions.txt, recipe.yaml, and verification.json for the saved environment, effective recipe, and validation details.
This is the prequantized Hugging Face checkpoint, not a compiled deployment package. Compile it with a compatible LLiMa release:
bash
llima-compile /path/to/Qwen3-VL-2B-Instruct-GPTQ-Safetensors -o /path/to/compiled-outputQuantization can reduce quality, especially for tasks unlike the text-only calibration data. The INT8 head improves the tested perplexity, visual accuracy, and repetition metrics over the INT4-head comparison, but does not resolve all reasoning, multi-turn memory, repetition, or image hallucination problems. This revision changes the tokenizer assets to the original upstream files; earlier repository revisions remain available through Hugging Face history. Validate the compiled model for the intended language, images, context length, and runtime.
hf download Qwen/Qwen3-VL-2B-Instruct \ --revision 89644892e4d85e24eaac8bacfd4f463576704203 \ --local-dir Qwen3-VL-2B-Instruct python quantize.py \ --model-path Qwen3-VL-2B-Instruct \ --output-dir Qwen3-VL-2B-Instruct-GPTQ-Safetensorsa5d85b6dcc535e6b93115a9ef287e6132fdbf30270da6218194ba742261173c7Python: 3.13.2CUDA: 12.8torch: 2.11.0+cu128transformers: 5.10.1llmcompressor: 0.12.0compressed-tensors: 0.17.1datasets: 5.0.0