kingjones777

Ornith-1.5-9B-Vision-NVFP4

README

License: apache-2.0

Why this build?

3.26× the decode of BF16 on the same box, same vLLM flags, exclusive GPU, same-prompt warm-ups
41.8% of the BF16 size — 7.33 GiB against 17.53 GiB
The output head is FP8, not 4-bit. With tie_word_embeddings: false this model has a real standalone lm_head; leaving it at 4 bits is a measurable quality risk, so it was excluded from the FP4 group and that exclusion was read back out of the file, not assumed
The vision tower is untouched (BF16) — this is a multimodal model and the projector is not quantization-damaged

What is actually quantized

Read from the model.safetensors header, 1362 tensors:

Table with columns: dtype, count, what
dtype	count	what
`U8`	200	NVFP4-packed Linear weights
`F8_E4M3`	201	FP8 tensors — the 200 activation scales plus `lm_head.weight`
`BF16`	559	vision tower, embeddings, norms
`F32`	402	scales

markdown
lm_head.weight                            F8_E4M3   [248320, 4096]
lm_head.weight_scale / input_scale        F32
model.language_model.embed_tokens.weight  BF16      [248320, 4096]

⚠️ embed_tokens is BF16 because it is an nn.Embedding — ModelOpt's recipe targets Linear modules, so an embedding is never a candidate. That is expected, not an omission.

Quick start

bash
hf download kingjones777/Ornith-1.5-9B-Vision-NVFP4 --local-dir ornith9b-nvfp4

bash
docker run --rm --ipc=host --gpus all -p 8000:8000 \
  -v $PWD/ornith9b-nvfp4:/model \
  vllm/vllm-openai:nightly-aarch64 \
  --model /model \
  --max-model-len 8192 --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.85 --max-num-seqs 1

Requires a vLLM build with ModelOpt NVFP4 kernels and Qwen3_5ForConditionalGeneration. The nightly aarch64 image above has both; older releases do not.

Measured — not estimated

Idle GPU, identical flags across arms, warm-ups on the same prompt as the samples, median of 5.

Table with columns: arm, 5 raw samples (tok/s), median, spread
arm	5 raw samples (tok/s)	median	spread
NVFP4 (this repo)	41.68, 41.66, 41.66, 41.90, 41.88	41.68	0.25
BF16 control	12.78, 12.77, 12.77, 12.76, 12.75	12.77	0.03

Correctness asserted on content + reasoning_content with finish_reason recorded, on every arm. Vision scored on spatial placement — a four-quadrant colour image where each quadrant is queried separately, so a fluent-but-misplaced answer fails.

⚖️ On calibration — what the evidence does and does not support

This checkpoint was produced with awq_clip on 384 real calibration samples (320 UltraChat conversations + 64 real Python files). An uncalibrated ablation (max amax over the same 384 samples, same FP8 head, same packing) was built and measured alongside it:

Table with columns: calibrated (this repo), plain ablation
	calibrated (this repo)	plain ablation
decode median	41.68	41.58
correctness	3/3	3/3
vision	4/4	4/4
exact-match vs BF16 (10 prompts)	0/10	0/10
mean Jaccard vs BF16	0.516	0.570

They are tied on every gate run here. The speed and size win comes from NVFP4 plus the FP8 head, not from the clip search — and this repository does not claim otherwise.

⚠️ That gate is three factual needles, a vision check and a 10-prompt divergence probe. It is a regression guard, not a capability benchmark. It shows the clip search did not separate from plain on these tests; it does not establish that calibration is worthless. Perplexity, KL divergence or a real benchmark suite might separate them, and none of those were run.

Not yet measured

Table with columns: Test, Status
Test	Status
Perplexity / KL divergence vs BF16	❓ not measured
MMLU-Pro, GPQA, GSM8K, HumanEval+	❓ not run
Context beyond 8192	❓ only 8192 served
Vision beyond quadrant placement (OCR, charts, documents)	❓ not measured
Throughput at batch > 1	❓ `--max-num-seqs 1` only
Non-GB10 Blackwell (B100/B200/RTX 50)	❓ untested
Independent reproduction	❓ none yet

Known issues

Needs a recent vLLM. Older builds lack either the ModelOpt NVFP4 kernels or the Qwen3_5ForConditionalGeneration architecture.
Reasoning parser did not auto-initialise on the tested image — reasoning_content came back empty and the reasoning text (including </think>) appeared inside content. Parse accordingly, or expect literal think tags in output.
No speculative decoding. config.json declares mtp_num_hidden_layers: 1, but the published safetensors contain no MTP weights — there is no draft head to load.
Single-file checkpoint, no index.json. Expected for an unsharded export, not a fault.

Source

ornith-ai/Ornith-1.5-9B, revision c927ad73. Architecture Qwen3_5ForConditionalGeneration — a hybrid stack (24 gated-deltanet + 8 full-attention layers), 32 layers, hidden 4096, vocab 248,320, tie_word_embeddings: false.

Independent results

None yet. If you run this build, please open a discussion with hardware, driver, vLLM version, exact flags, context, prompt-processing and generation tok/s, and peak memory. Independent reproductions will be listed separately from author benchmarks and carry more weight.

Acknowledgements

NVIDIA TensorRT Model Optimizer — the NVFP4 quantization and the HF checkpoint exporter used to produce this file.

vLLM — the serving runtime and its ModelOpt NVFP4 kernels; every number here was measured through it.

Ornith — the base model and its licence are theirs. This repository contributes quantization and measurement only.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

kingjones777

Model Tree

Base

ornith-ai/Ornith-1.5-9B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality