ironbcc

Qwen3-Embedding-4B-NVFP4

README

License: apache-2.0

Quantization details

python
QuantizationModifier(targets="Linear", scheme="NVFP4", ignore=["lm_head"])

Weights — FP4, group_size=16, strategy=tensor_group, symmetric, FP8 (e4m3) group scales.
Activations — FP4, group_size=16, dynamic="local" per-group scales with a static global FP8 scale fitted from calibration data. This is why the calibration set matters: W4A4 has a data-dependent activation scale, unlike a weights-only scheme.
Excluded — lm_head only. The model is dense (no MoE), so no router/gate exclusions apply. lm_head is unused by the pooling runner and is tied to the input embeddings.
Architecture is Qwen3ForCausalLM, so it loads with AutoModelForCausalLM; the embedding behaviour comes from last-token pooling, not a separate head.

Calibration

512 samples, max_seq_length=512, English only:

Table with columns: Source, Samples, Shape
Source	Samples	Shape
HuggingFaceH4/ultrachat_200k (MIT)	256	instruction / dialogue
google-research-datasets/natural_questions (CC BY-SA 3.0)	128	questions, carrying the query instruction prefix
natural_questions	128	passages, bare

Query-shaped samples carry the model's real query prefix (below) and passages do not, so the calibrated activation scales match the asymmetric way the model is used at inference.

Matryoshka patch — read this if you fork the recipe

Upstream Qwen3-Embedding-4B supports MRL truncation but its config.json declares neither is_matryoshka nor matryoshka_dimensions. vLLM refuses the dimensions pooling parameter unless ModelConfig.is_matryoshka is true, so this checkpoint adds:

json
{ "is_matryoshka": true,
  "matryoshka_dimensions": [128, 256, 512, 768, 1024, 1536, 2048, 2560] }

vLLM truncates after pooling and renormalizes, so a 768-dim response is unit-norm.

Serving with vLLM

bash
vllm serve ironbcc/Qwen3-Embedding-4B-NVFP4 \
  --served-model-name qwen3-embedding-4b \
  --runner pooling --convert embed \
  --pooler-config '{"dimensions": 768}' \
  --max-model-len 8192

--pooler-config '{"dimensions": 768}' makes every response 768-dim, so OpenAI-compatible clients need no dimensions parameter. Drop it to get the native 2560.

Measured on one Blackwell GPU at gpu_memory_utilization=0.09 (~8.8 GB resident): a 64-text batch (1654 prompt tokens) returns in 57-59 ms, about 0.9 ms/text.

Asymmetric use matters

Embed documents bare, and prefix queries with the string from the base model's config_sentence_transformers.json:

markdown
Instruct: Given a web search query, retrieve relevant passages that answer the query
Query: {your query}

This is not cosmetic. On a 31k-verse Bible corpus, "How can I be saved?" retrieves Psalm 55:16 / Genesis 49:18 unprefixed versus Acts 15:11 / Romans 10:10 / Titus 3:5 prefixed; "I am so lonely" goes from Job 7:19 to Psalm 25:16 / 102:7 / 88:8.

Official MTEB subset evaluation

Official MTEB 2.18.13 evaluators, five complete English test tasks (ArguAna, ArXivHierarchicalClusteringP2P, Banking77Classification.v2, BIOSSES, CQADupstackGamingRetrieval) — a targeted subset, not the full MTEB English v2 suite. Evaluated against the BF16 original and the W4A16 sibling, at every Matryoshka width used in production, same protocol for all three.

TL;DR: ~2.7GB weights (vs ~8GB BF16, ~3x smaller), weights and activations in FP4, at ~97-98% of BF16 MTEB score from 768 to 2048 dims. The W4A16 sibling trades the activation-speed gain for ~99% BF16 quality instead.

Average score across the 5 tasks:

Table with columns: Dim, BF16 (baseline), W4A16, W4A4 (this model)
Dim	BF16 (baseline)	W4A16	W4A4 (this model)
768	0.7329	0.7241 (98.8%)	0.7131 (97.3%)
1024	0.7335	0.7258 (99.0%)	0.7169 (97.7%)
1536	0.7340	0.7266 (99.0%)	0.7157 (97.5%)
2048	0.7351	0.7274 (99.0%)

Table with columns: Task, Dim, BF16, W4A16, W4A4
Task	Dim	BF16	W4A16	W4A4
ArguAna	768	0.7333	0.7114	0.6904
ArguAna	1024	0.7390	0.7159	0.6974
ArguAna	1536	0.7377	0.7150	0.6960
ArguAna

Retrieval tasks (ArguAna, CQADupstackGamingRetrieval) take the biggest hit under W4A4 (~4-5% relative on CQADupstackGamingRetrieval); clustering and classification stay close to BF16. Raw embedding cosine similarity vs BF16 (separate 100-text check, not a task score): mean ~0.95-0.96 across dims.

Matched protocol

Official MTEB task evaluators, complete test splits, dataset revisions pinned in the uploaded manifests.
64,880 encoded texts per arm; 24 concurrent request workers, batch size 32.
Retrieval instruction applied only to queries; documents remained bare.
Inputs longer than 8,192 tokens were right-truncated identically across all three arms.
MTEB cache disabled, isolated per-arm prediction artifacts, no result exceptions.

Raw scores: benchmarks/mteb_dims_comparison/all_scores.json.

Downstream check on the target application: with W4A4 vectors, dense retrieval over a 31,086-verse corpus returns John.3.16 as top-1 for "For God so loved the world" (cos 0.7865) and Psalm.23.1 for "the Lord is my shepherd" (0.7936).

Limitations

Reported MTEB results cover five English tasks, not the full English v2 or multilingual suites; do not treat them as a leaderboard aggregate.
BF16 baseline is now included (see above): W4A4 retains ~97-98% of BF16 average score across the tested widths, versus ~99% for W4A16.
W4A4 is lossy in a way that shows up directly in embedding space — see above. Validate on your own retrieval set before trusting it in production.
Blackwell/SM120-oriented. FP4 activation kernels need recent flashinfer; on vLLM 0.26.0 the flashinfer-python==0.6.14 pin has no matching flashinfer-cubin release, which may require FLASHINFER_DISABLE_VERSION_CHECK=1.
English calibration only. For multilingual retrieval, recalibrate on your languages.
Inherits the base model's Apache-2.0 license and its biases.

Provenance

Quantized with llm-compressor 0.12.0.1 in an environment isolated from the serving stack (transformers 5.10.1, torch 2.11.0+cu130). Conversion and calibration ran on a single Blackwell GPU. recipe.yaml and the oneshot_*.log in this repo are the actual artifacts emitted by the run.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

ironbcc

Model Tree

Base

Qwen/Qwen3-Embedding-4B

Quantized

this model

Input Modalities

Text

Output Modalities