Qwen3-Embedding-4B-W8A8 API & Inference Endpoint

Quantization

llm-compressor 0.14.0: SmoothQuant (strength 0.8) + GPTQ, scheme W8A8: weights int8 symmetric per-channel (static), activations int8 symmetric per-token (dynamic).
The last 4 decoder layers (32–35) are kept in bf16. They shape the last-token embedding and carried most of the GPTQ error; keeping them raised cosine to bf16 from 0.981 to 0.993 for ~0.4 GB.
Calibration: 256 samples, max 1024 tokens — Russian-language texts of a services marketplace (company and service descriptions, catalog rubrics, request templates, regions) as documents, plus search queries prefixed with the query instruction, roughly 3:1.

Check against bf16 (held-out samples, not used for calibration)

Table with columns: Metric, Value
Metric	Value
Cosine to bf16, mean / min (64 docs + 32 queries)	0.9929 / 0.9820
Top-5 overlap with bf16 ranking	0.8938

The numbers come from the quantized model in transformers (simulated quantization) right after calibration.

Usage (vLLM)

bash

vllm serve megafetis/Qwen3-Embedding-4B-W8A8 --runner pooling --served-model-name emb4# RTX 2080 Ti (no bf16): add --dtype half

Same as the original: last-token pooling, L2 normalization, 2560 dims (Matryoshka — can be truncated by the client). Queries need the instruction, documents do not:

text

Instruct: Given a web search query, retrieve relevant passages that answer the queryQuery:<query text>

Recipe — recipe.yaml; the scripts — ai-models/ of the project that produced it. License — Apache 2.0, as the base model.

Quantization

llm-compressor 0.14.0: SmoothQuant (strength 0.8) + GPTQ, scheme W8A8: weights int8 symmetric per-channel (static), activations int8 symmetric per-token (dynamic).
The last 4 decoder layers (32–35) are kept in bf16. They shape the last-token embedding and carried most of the GPTQ error; keeping them raised cosine to bf16 from 0.981 to 0.993 for ~0.4 GB.
Calibration: 256 samples, max 1024 tokens — Russian-language texts of a services marketplace (company and service descriptions, catalog rubrics, request templates, regions) as documents, plus search queries prefixed with the query instruction, roughly 3:1.

Check against bf16 (held-out samples, not used for calibration)

Table with columns: Metric, Value
Metric	Value
Cosine to bf16, mean / min (64 docs + 32 queries)	0.9929 / 0.9820
Top-5 overlap with bf16 ranking	0.8938

The numbers come from the quantized model in transformers (simulated quantization) right after calibration.

Usage (vLLM)

bash

vllm serve megafetis/Qwen3-Embedding-4B-W8A8 --runner pooling --served-model-name emb4# RTX 2080 Ti (no bf16): add --dtype half

Same as the original: last-token pooling, L2 normalization, 2560 dims (Matryoshka — can be truncated by the client). Queries need the instruction, documents do not:

text

Instruct: Given a web search query, retrieve relevant passages that answer the queryQuery:<query text>

Recipe — recipe.yaml; the scripts — ai-models/ of the project that produced it. License — Apache 2.0, as the base model.

Qwen3-Embedding-4B-W8A8

README

Quantization

Check against bf16 (held-out samples, not used for calibration)

Usage (vLLM)

Explore FriendliAI today

README

Quantization

Check against bf16 (held-out samples, not used for calibration)

Usage (vLLM)