- Quantization scheme: NVFP4 — weights in FP4 E2M1, scales in FP8 E4M3 per group of 16 elements
- Storage:
compressed-tensors (nvfp4-pack-quantized)
- Layers quantized: all
Linear layers except lm_head
- Original size (BF16): ~8 GB
- Quantized size: 2.7 GB (~3.4x compression)
Usage
This model is in compressed-tensors format. To load on a Blackwell GPU:
import torch
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained(
"andrebadini/Qwen3-Embedding-4B-NVFP4",
torch_dtype=torch.bfloat16,
trust_remote_code=True,
device_map="auto",
)
tok = AutoTokenizer.from_pretrained("andrebadini/Qwen3-Embedding-4B-NVFP4", trust_remote_code=True)
inputs = tok("Your text here", return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model(**inputs)
emb = outputs.last_hidden_state[:, -1, :]
print(emb.shape)
Inference server (vLLM)
Recommended (works out of the box on Blackwell):
vllm serve andrebadini/Qwen3-Embedding-4B-NVFP4 \
--runner pooling \
--convert embed \
--trust-remote-code
The --convert embed flag tells vLLM to run in embedding mode; the OpenAI-compatible
endpoint becomes POST /v1/embeddings.
Quantization recipe
default_stage:
default_modifiers:
QuantizationModifier:
targets: [Linear]
ignore: [lm_head]
scheme: NVFP4
Calibrated oneshot on 20 samples from neuralmagic/calibration (LLM split).
Hardware notes
- NVFP4 tensor cores require Blackwell (SM 120). On Hopper (H100) or older, kernels
will JIT-compile through CUTLASS via FlashInfer (slower first run, cached after).
- The flashinfer cache (
~/.cache/flashinfer/0.6.12/120f/cached_ops/fp4_gemm_cutlass_sm120/)
holds the JIT kernels; keep it across runs to avoid recompilation.
Limitations
- LM Studio does not currently load
compressed-tensors format. Use transformers directly
or run via vLLM / TGI.
- Other inference engines (llama.cpp, exllamav2, etc.) do not yet support NVFP4.
License
Inherits Apache 2.0 from the original Qwen3-Embedding-4B.
