Serving with vLLM
docker run --gpus all --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model 2uanDM/Qwen3.5-9B-FP8-Dynamic \ --max-model-len 20000 --gpu-memory-utilization 0.9 \ --max-num-seqs 128 --limit-mm-per-prompt.image 10
On a 20 GB card keep --max-num-seqs at ~200 or below: the hybrid linear-attention layers need one state slot per sequence,
and vLLM refuses to start if max_num_seqs exceeds the available slots.
Benchmarks
Hardware: NVIDIA RTX 4000 SFF Ada (20 GB, 70 W), vLLM 0.32.0, vllm bench serve with random prompts and --ignore-eos.
Baseline is the same model quantized in Qwen's block-wise FP8 format, which vLLM runs via Marlin (weight-only) on Ada.
Table with columns: Scenario, Block FP8 (Marlin, weight-only), This model (CUTLASS W8A8), Speedup| Scenario | Block FP8 (Marlin, weight-only) | This model (CUTLASS W8A8) | Speedup |
|---|
| Prefill-heavy — 4000 in / 16 out, concurrency 8: total tok/s | 2,179 | 4,308 | 1.98× |
| ↳ mean TTFT | 5,554 ms | 2,768 ms | 2.0× |
| Batched decode — 512 in / 256 out, concurrency 32: output tok/s | 395 | 472 | 1.20× |
| ↳ mean TTFT | 2,492 ms | 1,232 ms | 2.0× |
| Single user — 256 in / 256 out, concurrency 1: output tok/s | 27.4 | 26.0 | ~1.0× |
Single-stream decode is memory-bandwidth bound (both formats read 1 byte per weight), so the gain shows up in prefill and batched serving.
Reproduce with scripts/bench.sh.
Quality: sanity-checked only (arithmetic, factual QA and image description answer correctly). No formal eval has been run yet;
per-channel FP8 is typically near-lossless at this scale, but verify on your own tasks.
How it was made
scripts/fp8_dynamic_convert.py streams the original safetensors shards on CPU
(no GPU needed, ~20 GB RAM, a few minutes) and writes FP8 weights + per-channel scales with a compressed-tensors config:
python fp8_dynamic_convert.py /path/to/Qwen3.5-9B /path/to/Qwen3.5-9B-FP8-Dynamic
License
Apache-2.0, inherited from Qwen/Qwen3.5-9B. All credit for the model goes to the Qwen team.