shadowrock-io

Nemotron-3-Embed-8B-Community-FP8

README

License: openmdw-1.1

Benchmarks vs the unquantized model

Comparison column = NVIDIA's official per-task results from the mteb results repo — their numbers, not our reproduction. Our runs: mteb 2.18.12, vLLM 0.26.0, mean pooling, query: /passage: prefixes, max length 8192 (NVIDIA evaluated at 4096; on these tasks few documents exceed either limit). Tasks are the open (public) RTEB datasets in the domains where the base model ranks top-4 on the RTEB leaderboard: Finance #1, German #1, Code #2, Healthcare #4, Legal #4.

Table with columns: Task (nDCG@10), NVIDIA official BF16, FP8 (this repo), Delta
Task (nDCG@10)	NVIDIA official BF16	FP8 (this repo)	Delta
HumanEvalRetrieval	1.0000	1.0000	±0.0000
MBPPRetrieval	0.9560	0.9564	+0.0004
WikiSQLRetrieval	0.9950	0.9947	−0.0003
DS1000Retrieval	0.7646	0.7626	−0.0020
FinanceBenchRetrieval	0.9526	0.9532	+0.0006
HC3FinanceRetrieval	0.7981	0.7995	+0.0014
FinQARetrieval	0.8871	0.8841	−0.0030
LegalQuAD (German)	0.7718	0.7638	−0.0080
LegalSummarization	0.7666	0.7660	−0.0006
ChatDoctorRetrieval	0.7690	0.7676	−0.0014
AILAStatutes	0.5826	0.5782	−0.0044
AILACasedocs	0.4942	0.4890	−0.0052

Mean delta −0.0019 across all 12 tasks; −0.0013 on the 10-task subset shared by all three community builds (the two AILA legal tasks were run only on the CUDA builds). The private RTEB datasets can only be run by the MTEB team, so this table covers the open subset.

Regression vs our own BF16 baseline (identical harness both sides)

BF16 baseline computed with the same code, adapter, prefixes, and pins on an A100. Gate: per-task nDCG@10 loss ≤ 0.01.

Table with columns: Task, BF16, FP8, Delta
Task	BF16	FP8	Delta
NFCorpus	0.4237	0.4220	−0.0016
SciFact	0.8330	0.8328	−0.0002
FiQA2018	0.6564	0.6568	+0.0004
ArguAna	0.6314	0.6330	+0.0016
TRECCOVID

All pass. Embedding-level fidelity vs BF16 on token-ID-locked fixtures: cosine 0.9967–0.9979 (runtime kernels), matching the ModelOpt fake-quant simulation (0.9982–0.9990). Raw result JSON ships under results/.

Serving with vLLM

python
from vllm import LLM
from vllm.config import PoolerConfig

llm = LLM(
    model="shadowrock-io/Nemotron-3-Embed-8B-Community-FP8",
    runner="pooling",
    pooler_config=PoolerConfig(seq_pooling_type="MEAN"),  # default LAST is silently wrong
    max_model_len=8192,
)
out = llm.embed(["query: what does FP8 change?", "passage: Only the weight format."])

Required patch for vLLM ≤ 0.26.0: vLLM's pooling adapter replaces the checkpoint's absent lm_head with a placeholder layer, and ModelOptFp8LinearMethod.process_weights_after_loading crashes on the placeholder's meta tensors (Tensor.item() cannot be called on meta tensors). Run scripts/patch_modelopt_fp8_guard.py once against your vLLM install before loading (idempotent; an upstream fix has been proposed).

Notes that matter for correct embeddings:

Pooling must be MEAN and attention is bidirectional; both come from the checkpoint config, but the pooler override above guards against defaults.
Prefixes are your job: query: / passage: . The server does not add them.
Texts longer than max_model_len are rejected by vLLM's pooling runner — truncate at the tokenizer (truncation=True, max_length=8192) and pass token IDs.
Embeddings are 4096-dim; L2-normalize before use. Matryoshka truncation (2048/1024): slice, then re-normalize.
On pre-Ada GPUs (SM < 89, e.g. A100) vLLM falls back to weight-only Marlin kernels — functional, but not the W8A8 path measured here.

Measured on: NVIDIA H200 (validation runs), GeForce RTX 5070 Ti (SM120, serving validation), vLLM 0.26.0, CUDA 12.8.

Quantization details

Method: NVIDIA TensorRT Model Optimizer (ModelOpt) FP8 post-training quantization — E4M3 weights with per-tensor activation scales; embeddings, norms, and pooling untouched. Full module inventory: quantization/module_inventory.json.
Derived in a fresh process from the pinned BF16 snapshot — never from another quantized model object.
Calibration: ~1k public samples from MS MARCO and MIRACL train splits, token-bucketed (32–16k tokens) with real prefix distribution. MS MARCO is research-licensed, so the manifest ships dataset IDs + a deterministic builder script, not text. Eval-set contamination audit (by ID and content hash) included.
Quantize/eval scripts ship under scripts/; raw eval JSON under results/.

Caveats

MIRACL multilingual coverage in the regression suite is two held-out languages (Swahili, Telugu, hard-negatives variants) on the NVFP4 companion; this FP8 build's multilingual evidence is LegalQuAD (German) plus the base model's own multilingual results. Full-corpus MIRACL was excluded for compute cost.
NVIDIA evaluated at sequence length 4096; our runs use 8192. On the tasks above the difference is immaterial (few documents exceed 4096 tokens), but it is a protocol difference.

Intended use & limitations

Intended uses are the base model's: dense retrieval, semantic search, and RAG indexing over text corpora, with query: /passage: prefixed inputs. The base card's intended-use, safety, and language-coverage statements — nvidia/Nemotron-3-Embed-8B-BF16 — carry over unchanged; quantization alters none of the model's behavior boundaries, only its numeric precision. Our evaluation establishes parity on the benchmarks listed above and nothing beyond them: other languages, domains, sequence-length regimes, and hardware/runtime combinations inherit the base model's behavior with quantization noise that we have not measured there.

Attribution & citation

Quantization, validation harness, and card by Matt Busi (@mattbusi on Hugging Face) at ShadowRock. If you use this build, cite the NVIDIA base model — the embedding quality is theirs:

bibtex
@misc{nvidia2026nemotron3embed,
  title  = {Nemotron-3-Embed-8B},
  author = {NVIDIA},
  year   = {2026},
  url    = {https://huggingface.co/nvidia/Nemotron-3-Embed-8B-BF16}
}

License

OpenMDW-1.1, inherited from the base model (see LICENSE). NOTICE carries the upstream Apache-2.0 attribution for the Ministral component plus our modification statement. Community build by ShadowRock; no NVIDIA affiliation or endorsement.

About ShadowRock

ShadowRock is an AI-specialized systems integrator and Zendesk Premier Partner. We help businesses get real value from their go-to-market technology, from CRM and support platforms to applied AI like the models in this collection. Find us at shadowrock.io or on LinkedIn.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

shadowrock-io

Model Tree

Base

nvidia/Nemotron-3-Embed-8B-BF16

Quantized

this model

Input Modalities

Text

Output Modalities