Model Overview
- Model Architecture: Qwen/Qwen3.5-9B (Qwen3_5ForConditionalGeneration)
- Input: Text / Image
- Output: Text
- Model Optimizations:
- Weight quantization: FP4 (NVFP4)
- Activation quantization: FP4 (NVFP4)
- Release Date: 11/06/2026
- Version: 1.1
- Model Developers: RedHatAI
This model is a quantized version of Qwen/Qwen3.5-9B.
It was evaluated on several tasks to assess its quality.
Model Optimizations
This model was obtained by quantizing the weights and activations of the language-model linear
operators of Qwen/Qwen3.5-9B to FP4 (NVFP4) data type,
ready for inference with vLLM. The vision encoder and the MTP layer are kept in BF16, matching the
source checkpoint.
This optimization reduces the number of bits per parameter in the quantized layers from 16 to 4,
roughly halving the checkpoint size (9.4 GB vs 19.3 GB BF16).
Quantization was performed with LLM Compressor
using GPTQ with the NVFP4 scheme and nvfp4_expanded_imatrix weight observer, block size 128,
static actorder and dampening fraction 0.01; recipe.yaml in this repository holds the exact
recipe. The model is exported in compressed-tensors format.
Deployment
Use with vLLM
The model runs on a single GPU. Following the official
vLLM recipe:
vllm serve RedHatAI/Qwen3.5-9B-NVFP4 \ --tensor-parallel-size 1 \ --max-model-len 262144 \ --reasoning-parser qwen3
Tool use:
vllm serve RedHatAI/Qwen3.5-9B-NVFP4 \ --tensor-parallel-size 1 \ --max-model-len 262144 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder
The checkpoint ships the MTP weights, so vLLM's MTP speculative decoding can be enabled with
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'.
Evaluation
This model and the unquantized Qwen/Qwen3.5-9B were
evaluated on GSM8K, GPQA Diamond, MathVista (testmini split) and MMLU-Pro using
Inspect with the inspect_evals task suite, served
with vLLM (OpenAI-compatible API, single B200 GPU, vLLM nightly). Both models ran through the
identical harness, serving stack, sampling and limits — single epoch (pass@1), thinking mode
enabled, sampling following the Qwen3.5 card's thinking-mode recommendation (temperature 1.0,
top_p 0.95, top_k 20, presence_penalty 1.5) — so the recovery column below is a controlled
like-for-like comparison.
Accuracy
Standard errors of the measured scores: GSM8K ±0.6 (BF16) / ±0.6 (NVFP4), GPQA Diamond ±2.7 / ±2.9,
MathVista ±1.1 / ±1.1, MMLU-Pro ±0.3 / ±0.4 percentage points. The quantized checkpoint reasons
longer than the BF16 reference (mean output on MMLU-Pro 6,455 vs 5,029 tokens; 158 of 12,032
samples reached the 32,768-token generation cap, scoring as failures), which accounts for most of
its remaining losses. For reference, the
Qwen3.5-9B model card reports 81.7 (GPQA Diamond), 85.7
(MathVista mini) and 82.5 (MMLU-Pro) under Qwen's own protocol, consistent with the BF16
measurements above.
Reproduction
Both checkpoints (this model and Qwen/Qwen3.5-9B) were served with vLLM and evaluated for one
epoch per benchmark with identical settings (only the model id differs):
vllm serve RedHatAI/Qwen3.5-9B-NVFP4 \ --tensor-parallel-size 1 \ --max-model-len 262144 \ --reasoning-parser qwen3 \ --default-chat-template-kwargs '{"enable_thinking": true}'
COMMON="--model vllm/RedHatAI/Qwen3.5-9B-NVFP4 --temperature 1.0 --top-p 0.95 --presence-penalty 1.5 --generate-config generate_config.json"# generate_config.json: {"extra_body": {"top_k": 20, "chat_template_kwargs": {"enable_thinking": true}}} inspect eval inspect_evals/gsm8k $COMMON --max-tokens 32768inspect eval inspect_evals/mmlu_pro $COMMON --max-tokens 32768inspect eval inspect_evals/gpqa_diamond $COMMON --max-tokens 81920inspect eval inspect_evals/mathvista $COMMON --max-tokens 81920