What was changed
- Language model linear layers. Weights are int4, symmetric, per channel; activations are int8, dynamic, per token (W4A8). On SM89
this runs on the Humming INT8 tensor-core path.
- Rotation and GPTQ. Each linear layer's input gets a Hadamard rotation (QuIP style, block 128). vLLM applies the inverse online through
hadacore.
GPTQ with static act-order runs after the rotation. The recipe is in recipe.yaml.
- Kept in bf16. The vision tower (all blocks,
merger, deepstack_merger_list, patch_embed, pos_embed), embed_tokens and all norms.
- No
lm_head. The original checkpoint does not have one, and pooling does not need it.
- Tokenizer, processor and chat template. These are unchanged from the original.
The rotation and GPTQ are required. Plain round-to-nearest W4A8 breaks image embeddings (text→image R@1 falls from 0.68 to 0.03).
W4A8 with GPTQ but no rotation gets 0.57.
Precision by layer
The language model has 36 decoder layers, all with full attention and a dense MLP.
Weights stored in the checkpoint:
Table with columns: module, layers, weight dtype, weight scale, activation dtype, Hadamard| module | layers | weight dtype | weight scale | activation dtype | Hadamard |
|---|
self_attn.q_proj, k_proj, v_proj, o_proj | all 36 | INT4 symmetric, per channel | BF16 per channel | INT8 dynamic, per token | yes |
mlp.gate_proj, up_proj, down_proj | all 36 | INT4 symmetric, per channel | BF16 per channel | INT8 dynamic, per token | yes |
input_layernorm, post_attention_layernorm, q_norm, k_norm, final norm | all | BF16 | — | BF16 | — |
embed_tokens | — | BF16 | — | — | — |
vision tower: 27 blocks, merger, 3 deepstack_merger_list, patch_embed, pos_embed | — | BF16 | — | BF16 | — |
The checkpoint has no FP8 or FP4 weights. The Hadamard matrices (128×128) are stored in FLOAT64 as *.v_input.weight.
Created at runtime by the flags in "How to run":
Table with columns: what, dtype| what | dtype |
|---|
KV cache, --kv-cache-dtype fp8 | FP8 E4M3, per-tensor scale 1.0 (no calibrated scales in the checkpoint) |
| rotated layer input before int8 quantization | BF16 |
| pooled embedding (last token, L2-normalized) | FP32 in the response |
Calibration datasets
The calibration set has 512 samples of up to 2048 tokens. Every item used for evaluation was excluded.
No private data was used.
How to run
Requirements:
- A vLLM build with the Humming linear backend and the
hadacore transform. It was tested on the build above; older releases do not load the rotated W4A8 layout. The exact build is the public image vllm/vllm-openai:nightly-cf5cea2da37d3dfed27d2e0aa3041c3e553c2d57.
- An NVIDIA GPU with int8 tensor cores. Only SM89 (RTX 6000 Ada) was tested. On SM89, an fp8 KV cache works only with FlashInfer.
vllm serve shabonasar/qwen3-vl-embedding-8b-w4a8-hadamard-ada \ --runner pooling \ --hf-overrides '{"is_matryoshka": true}' \ --linear-backend humming \ --attention-backend FLASHINFER \ --kv-cache-dtype fp8 \ --max-model-len 8192 \ --max-num-batched-tokens 8192 \ --limit-mm-per-prompt '{"image": 1, "video": 0}' \ --gpu-memory-utilization 0.35
To embed text, call POST /v1/embeddings with {"input": ["..."], "dimensions": 1024}. dimensions is optional. To embed an image:
{"messages": [{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}]}], "add_generation_prompt": true}
Measurements (single RTX 6000 Ada 48 GB, vLLM commit cf5cea2da)
The comparison is FP8 W8A8 dynamic on the vLLM cutlass kernel with FlashAttention 2 and bf16 KV, the fastest FP8 setup on this card.
--gpu-memory-utilization 0.40 was used for this model and 0.35 for FP8. Prefix caching was off and every input was unique.
The model has no decode phase, so all throughput is prefill. There is no speculative decoding.
Table with columns: workload, FP8 W8A8, this model| workload | FP8 W8A8 | this model |
|---|
| prefill, 128-token inputs, 32 parallel, tok/s | 14 814 | 20 997 |
| prefill, 512-token inputs, 8 parallel, tok/s | 14 243 | 20 850 |
| prefill, 2048-token inputs, 8 parallel, tok/s | 13 397 | 18 338 |
| prefill, 8000-token inputs, 8 parallel, tok/s | 11 413 | 15 190 |
| prefill, one batch of 64 × 512 tokens, tok/s | 13 668 | 18 718 |
| latency of one request, 128 / 512 / 2048 / 8000 tokens, ms | 17.8 / 39.0 / 149 / 704 | 13.8 / 33.3 / 106 / 527 |
| images, 8 parallel, img/s | 62.9 | 82.6 |
| latency of one image, ms | 48.6 | 61.7 |
| KV cache capacity, tokens | 44 032 | 164 864 |
Single-image latency is slightly higher because of the online rotation in the image-token prefill, but image throughput under load is 1.3× higher.
Quality on held-out sets: RuBQ (300 Russian queries against 3000 passages) and Flickr8k (300 images and their captions).
Table with columns: RuBQ nDCG@10, text→image R@1, image→text R@1 | RuBQ nDCG@10 | text→image R@1 | image→text R@1 |
|---|
| bf16 original | 0.947 | 0.683 | 0.710 |
| FP8 W8A8 dynamic | 0.945 | 0.707 | 0.707 |
| W4A8, round-to-nearest | 0.922 | 0.027 | 0.313 |
| W4A8 + GPTQ, no rotation | 0.940 | 0.567 | 0.637 |
| this model, fp8 KV | 0.952 | 0.697 | 0.707 |
| this model, nvfp4 KV | 0.948 | 0.583 | 0.680 |
Mean cosine similarity to the bf16 embeddings is 0.98 for text and 0.86 for images. The retrieval metrics stay at bf16 level, but vectors from this
model should not be mixed with vectors from the original in the same index.
Limitations
- It was tested only on SM89 with the vLLM build above.
- Image embeddings differ more from bf16 than text embeddings (cosine 0.86), so re-index image collections built with another version.
- An NVFP4 KV cache costs 11 points of text→image R@1. Use fp8.
- The numbers above come from our own test sets on one GPU model. Check them on your workload.
License
Apache-2.0, the same as the base model Qwen/Qwen3-VL-Embedding-8B. The calibration
datasets were used only for calibration and are not redistributed. See their pages for their own terms.