Model Details
Table with columns: Field, Value| Field | Value |
|---|
| Base model | deepreinforce-ai/Ornith-1.0-35B |
| Base revision | 5df2ed3f675c7beaa490328cc70bb573b65fb660 |
| Release repo | LS-ML/Ornith-1.0-35B-ModelOpt-NVFP4-Expert |
| Architecture | Qwen3_5MoeForConditionalGeneration |
| Model type | qwen3_5_moe |
| Modality | Text and vision-language |
| Max context tested | 262144 tokens |
| Quantization | ModelOpt NVFP4, experts-only target |
| Group size | 16 |
| Exported checkpoint size | 23G on disk |
| Source BF16 checkpoint size | 66G on disk |
| License | MIT |
The repository name uses Expert, but the quantization target is experts-only:
the MoE expert linear weights are quantized while embeddings, lm head, visual
modules, attention, linear-attention modules, and shared experts are excluded.
See hf_quant_config.json and config.json for the exact ModelOpt
quantization metadata.
Benchmark Summary
The checkpoint was benchmarked against the upstream BF16 model on the same DGX
Spark / GB10, same vLLM nightly container, same 262K serving profile, same FP8
KV cache allocation, and same benchmark harness.
Performance summary:
Table with columns: Metric, BF16, ModelOpt NVFP4, Result| Metric | BF16 | ModelOpt NVFP4 | Result |
|---|
| Model directory size | 66G | 23G | 2.9x smaller |
| vLLM model loading memory | 65.53 GiB | 22.41 GiB | 2.9x lower |
| Weight loading time | |
Accuracy summary:
Table with columns: Benchmark, BF16 acc / score, NVFP4 acc / score, Delta NVFP4-BF16| Benchmark | BF16 acc / score | NVFP4 acc / score | Delta NVFP4-BF16 |
|---|
| MMLU-Pro | 55.8% | 55.0% | -0.8 pp |
| GPQA Diamond mirror | 22.7% | 13.6% | -9.1 pp |
| MATH-500 |
Detailed performance and accuracy benchmark tables are included below.
Benchmark Graphics
Performance benchmark:
Accuracy and quality benchmark:
Quantization
The checkpoint was produced with NVIDIA ModelOpt from a fused-expert staging copy
of the upstream BF16 model. The upstream checkpoint stores split expert tensors;
the staging copy fused expert gate/up/down tensors into the layout expected by
current Transformers and ModelOpt. The upstream BF16 checkpoint itself was not
modified.
Quantization metadata:
Table with columns: Field, Value| Field | Value |
|---|
| ModelOpt source version | 0.46.0.dev106+g6cc522658 |
| ModelOpt commit | 6cc5226588f0668679df03ba4646b7dfec32f99c |
| Quant algorithm | NVFP4 |
| KV cache quantization during export | none |
| Calibration dataset | nvidia/Nemotron-SFT-Agentic-v2, split search |
| Calibration settings | , , |
The quantization notes used during export are included in
QUANTIZATION_NOTES.md.
Serving With vLLM
Validated serving stack:
vllm/vllm-openai:nightly- ModelOpt quantization loader:
--quantization modelopt
- FlashInfer attention backend
- Blackwell / GB10 tested with
CUTE_DSL_ARCH=sm_121a
- OpenAI-compatible chat completions
- Text, image, and Qwen3 XML tool-call smoke tests
Example full-context DGX Spark / GB10 launch:
docker run --rm \
--name ornith35-nvfp4-vllm \
--gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --shm-size=32g \
-p 8000:8000 \
-v /path/to/models:/models \
-v ~/.cache:/root/.cache \
-e FLASHINFER_DISABLE_VERSION_CHECK=1 \
-e CUTE_DSL_ARCH=sm_121a \
vllm/vllm-openai:nightly \
--model /models/Ornith-1.0-35B-ModelOpt-NVFP4-Expert \
--host 0.0.0.0 \
--port 8000 \
--served-model-name ornith35-nvfp4 \
--trust-remote-code \
--dtype bfloat16 \
--quantization modelopt \
--kv-cache-dtype fp8 \
--kv-cache-memory-bytes 28G \
--attention-backend flashinfer \
--max-model-len 262144 \
--max-num-seqs 4 \
--max-num-batched-tokens 8192 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--enable-chunked-prefill
Lower-memory smoke profile:
docker run --rm \
--name ornith35-nvfp4-vllm-smoke \
--gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --shm-size=32g \
-p 8000:8000 \
-v /path/to/models:/models \
-v ~/.cache:/root/.cache \
-e FLASHINFER_DISABLE_VERSION_CHECK=1 \
-e CUTE_DSL_ARCH=sm_121a \
vllm/vllm-openai:nightly \
--model /models/Ornith-1.0-35B-ModelOpt-NVFP4-Expert \
--host 0.0.0.0 \
--port 8000 \
--served-model-name ornith35-nvfp4 \
--trust-remote-code \
--dtype bfloat16 \
--quantization modelopt \
--kv-cache-dtype fp8 \
--kv-cache-memory-bytes 4G \
--attention-backend flashinfer \
--max-model-len 8192 \
--max-num-seqs 1 \
--max-num-batched-tokens 8192 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--enable-chunked-prefill
For non-thinking deterministic chat/eval requests, use:
{
"temperature": 0,
"chat_template_kwargs": {
"enable_thinking": false
}
}
Validation
The quantized checkpoint was validated on a local DGX Spark / GB10 against the
BF16 upstream checkpoint using the same vLLM runtime profile, same maximum
context, same FP8 KV cache allocation, and the same benchmark harness.
All text rows below used unique prompts, prefix caching disabled, max_tokens=128,
and submitted concurrency 1, 2, or 4.
Table with columns: Context, Conc, BF16 prefill, NVFP4 prefill, Prefill speedup, BF16 decode, NVFP4 decode, Decode speedup, BF16 TTFT, NVFP4 TTFT| Context | Conc | BF16 prefill | NVFP4 prefill | Prefill speedup | BF16 decode | NVFP4 decode | Decode speedup | BF16 TTFT | NVFP4 TTFT |
|---|
| 32k | 1 | 3288.8 | 4626.2 | 1.41x | 29.6 | 35.5 | 1.20x | 10.0s | 7.1s |
Throughput units are tokens per second. TTFT is max time to first token for the
concurrent batch.
The serial 4096 x 4096 image test:
Table with columns: Case, BF16 prefill, NVFP4 prefill, Prefill speedup, BF16 decode, NVFP4 decode, Decode speedup, BF16 TTFT, NVFP4 TTFT| Case | BF16 prefill | NVFP4 prefill | Prefill speedup | BF16 decode | NVFP4 decode | Decode speedup | BF16 TTFT | NVFP4 TTFT |
|---|
| 4K image | 1204.6 | 1357.6 | 1.13x | 30.0 | 36.1 | 1.20x | 13.6s | 12.1s |
Startup and memory observations:
Table with columns: Metric, BF16, ModelOpt NVFP4| Metric | BF16 | ModelOpt NVFP4 |
|---|
| Remote model directory size | 66G | 23G |
| vLLM model loading memory | 65.53 GiB | 22.41 GiB |
| Weight loading time | 454.55s | 153.13s |
| Engine init observed to health |
Accuracy Benchmark Results
Quality runs used deterministic decoding with temperature=0,
chat_template_kwargs={"enable_thinking": false}, and paired per-item scoring.
The key quantization regression count is "BF16 correct / NVFP4 wrong".
Table with columns: Benchmark, Items, BF16 acc, NVFP4 acc, Delta NVFP4-BF16, BF16 correct/NVFP4 wrong, NVFP4 correct/BF16 wrong| Benchmark | Items | BF16 acc | NVFP4 acc | Delta NVFP4-BF16 | BF16 correct/NVFP4 wrong | NVFP4 correct/BF16 wrong |
|---|
| MMLU-Pro | 12032 | 55.8% | 55.0% | -0.8 pp | 527 | 429 |
| GPQA Diamond mirror | 198 | 22.7% | 13.6% | -9.1 pp | 23 |
ToolEvalBench version 2.0.7 was run sequentially over 69 standard scenarios:
Table with columns: Model, Final score, Points, Deployability, Responsiveness, Safety warnings| Model | Final score | Points | Deployability | Responsiveness | Safety warnings |
|---|
| BF16 | 82 | 113 / 138 | 74 | 54 | 3 |
| NVFP4 | 88 | 121 / 138 | 81 | 64 | 1 |
Interpretation: this NVFP4 export is close to BF16 on broad multiple-choice,
multimodal, OCR, and MBPP-style code tasks, but it showed clear regressions on
GPQA Diamond and HumanEval+ in this run.
Caveats
- This is a community quantization, not an official upstream release.
- The checkpoint has been validated with vLLM ModelOpt loading. Other loaders
may not support this ModelOpt NVFP4 format.
- vLLM marks ModelOpt NVFP4 support as experimental, so revalidate after major
vLLM, ModelOpt, CUDA, or FlashInfer changes.
- The export does not include FP8 KV q/prob scaling factors. When serving with
--kv-cache-dtype fp8, vLLM reports that it uses scale 1.0; treat this as
a quality caveat for accuracy-sensitive workloads.
- GPQA used the ungated
fingertap/GPQA-Diamond mirror because the official
dataset was gated at the time of evaluation.
- HumanEval+ and MBPP+ used EvalPlus prompts and expanded inputs, but scoring
used a local subprocess checker because the official EvalPlus sandbox failed
on the local macOS host with resource-limit errors.
- BFCL v3 10-each should be treated as raw paired signal only; ToolEvalBench is
the stronger tool-use benchmark in this report.
Responsible Use
Use this model consistently with the upstream model license and any applicable
laws or platform policies. Because this is a quantized derivative, evaluate it
for your own target domain before relying on it in production or
accuracy-sensitive workflows.
License And Attribution
The upstream model is MIT licensed. This quantized release preserves the MIT
license and attribution to deepreinforce-ai/Ornith-1.0-35B.
Upstream model:
https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B
Quantized by LS-ML.