primitive-ai

Ornith-1.5-35B-A3B-agentic-NVFP4-FP8

Which of the two should you use

Read this before choosing, because the honest answer is narrower than a headline number.

Table with columns: tool calling, knowledge, pooled, tool-calling sd
	tool calling	knowledge	pooled	tool-calling sd
this repo	73.8 (n=5)	90.9 (n=3)	88.4	1.2
our general build	71.8 (n=6)	91.7	88.8	3.1
BF16 reference	75.1 (n=4)	92.0	89.5	1.4

⚖️ On tool-calling accuracy this build is level with BF16 — 73.8 against 75.1, a difference this suite cannot resolve (t=1.5). Its mean settled downward as we added runs (75.3 at n=2, 74.1 at n=4, 73.8 at n=5), which is why the number here is a five-run mean and not a screening result.
🤔 It is not provably better than our general build at tool calling. The gap is +2.1 points at t=1.5 — suggestive, not established. We are not going to claim it.
📉 What it demonstrably is, is steadier — on every sub-metric. Against the general build: tool-calling sd 1.2 vs 3.1, calling sd 1.1 vs 3.7, abstention sd 2.9 vs 4.9. Six runs of the general build span 67.5 to 75.5; five runs of this one span 72.5 to 75.5. If a model sits behind an agent, a build whose tool-calling accuracy moves eight points between identical runs is harder to operate than one a point lower and stable. That reproducibility is the reason this artifact exists.
📚 It costs 0.8 points of general knowledge (90.9 vs 91.7), and on the pooled score across both suites the general build is 0.3 ahead (88.8 vs 88.5). There is no free lunch here.
🎯 Where the gain actually sits: calling, not declining. On the 160 items that require a call it scores 78.1 ± 0.5 against the general build's 75.1 ± 1.5. On the 40 items whose right answer is to call nothing it is marginally worse — 56.5 against 58.3, inside the noise. An earlier version of this card claimed the opposite ("declines better, 60.0 vs 55.0"); that was two single runs, and it did not survive five.

So: take this build if you value reproducible tool-calling behaviour and can spare 0.8 points of knowledge. Take the general build for question answering, or if you want the best pooled score.

Serve it

bash
vllm serve primitive-ai/Ornith-1.5-35B-A3B-agentic-NVFP4-FP8 --max-model-len 32768

Verified to load on Blackwell (native NVFP4/FP8) and on Ampere (A100, weight-only Marlin path — the BF16 lm_head is what keeps that path loadable).

Measured

1,370 items across fourteen public benchmarks — a 1,170-item knowledge suite over nine of them and a 200-item tool-calling suite over five more (BFCL v4, xLAM/APIGen, ToolACE, Glaive v2, nvidia When2Call). One fixed protocol for every row: temperature 0.6 / top_p 0.95 / top_k 20, thinking forced on, a 16,384-token budget, concurrency 32, on one RTX PRO 6000 Blackwell.

Table with columns: build, size, tool calling, knowledge, pooled
build	size	tool calling	knowledge	pooled
BF16 reference	67.0 G	75.1 ± 0.7 (n=4)	92.0	89.5
ornith-ai FP8	36.7 G	74.9 ± 0.8 (n=4)	91.2	89.0
this repo	22.6 G	73.8 ± 0.5 (n=5)	90.9 (n=3)	88.4

None of the four quants above is statistically distinguishable from another on tool calling. The whole column spans 2.3 points against per-build standard deviations of 0.6 to 1.3. Treat it as a tie and choose on size, knowledge, and — the axis this build is for — consistency.

Every tool-calling figure above is a mean of repeated runs, with its n. That matters more on this model than on any other we have measured: a single run of the general build can land anywhere between 67.5 and 75.5. We published single-run numbers for this suite once and they were misleading — the quantized builds of this model carry two to three times BF16's run-to-run spread, so one run cannot rank them. Differences under about 2 points here are not differences.

abstain is the other half of the story: on the 40 items whose correct action is to call nothing, this build scores 60.0 against the general build's 55.0. Calibrating on tool-call traffic makes a model more conservative about invoking tools — better at declining, marginally less aggressive at calling — which is usually what you want from an agent.

What's quantized to what

Table with columns: tensors, format
tensors	format
all 40 layers' routed experts (`gate/up/down_proj`)	NVFP4 (group 16)
`self_attn` q/k/v/o, `linear_attn` in/out projections, shared-expert projections	FP8 E4M3
`lm_head`, embeddings, vision tower, MTP block, router gates, norms	BF16

Rounding is chosen by GPTQ against a tool-call and agent-trajectory calibration set that is disjoint from the evaluation suite — rows sharing a prompt with any eval item were removed before calibration, so the numbers above are measurements rather than fits.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

primitive-ai

Model Tree

Base

ornith-ai/Ornith-1.5-35B-A3B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Which of the two should you use

Read this before choosing, because the honest answer is narrower than a headline number.

Table with columns: tool calling, knowledge, pooled, tool-calling sd
	tool calling	knowledge	pooled	tool-calling sd
this repo	73.8 (n=5)	90.9 (n=3)	88.4	1.2
our general build	71.8 (n=6)	91.7	88.8	3.1
BF16 reference	75.1 (n=4)	92.0	89.5	1.4

⚖️ On tool-calling accuracy this build is level with BF16 — 73.8 against 75.1, a difference this suite cannot resolve (t=1.5). Its mean settled downward as we added runs (75.3 at n=2, 74.1 at n=4, 73.8 at n=5), which is why the number here is a five-run mean and not a screening result.
🤔 It is not provably better than our general build at tool calling. The gap is +2.1 points at t=1.5 — suggestive, not established. We are not going to claim it.
📉 What it demonstrably is, is steadier — on every sub-metric. Against the general build: tool-calling sd 1.2 vs 3.1, calling sd 1.1 vs 3.7, abstention sd 2.9 vs 4.9. Six runs of the general build span 67.5 to 75.5; five runs of this one span 72.5 to 75.5. If a model sits behind an agent, a build whose tool-calling accuracy moves eight points between identical runs is harder to operate than one a point lower and stable. That reproducibility is the reason this artifact exists.
📚 It costs 0.8 points of general knowledge (90.9 vs 91.7), and on the pooled score across both suites the general build is 0.3 ahead (88.8 vs 88.5). There is no free lunch here.
🎯 Where the gain actually sits: calling, not declining. On the 160 items that require a call it scores 78.1 ± 0.5 against the general build's 75.1 ± 1.5. On the 40 items whose right answer is to call nothing it is marginally worse — 56.5 against 58.3, inside the noise. An earlier version of this card claimed the opposite ("declines better, 60.0 vs 55.0"); that was two single runs, and it did not survive five.

So: take this build if you value reproducible tool-calling behaviour and can spare 0.8 points of knowledge. Take the general build for question answering, or if you want the best pooled score.

Serve it

bash
vllm serve primitive-ai/Ornith-1.5-35B-A3B-agentic-NVFP4-FP8 --max-model-len 32768

Verified to load on Blackwell (native NVFP4/FP8) and on Ampere (A100, weight-only Marlin path — the BF16 lm_head is what keeps that path loadable).

Measured

Table with columns: build, size, tool calling, knowledge, pooled
build	size	tool calling	knowledge	pooled
BF16 reference	67.0 G	75.1 ± 0.7 (n=4)	92.0	89.5
ornith-ai FP8	36.7 G	74.9 ± 0.8 (n=4)	91.2	89.0
this repo	22.6 G	73.8 ± 0.5 (n=5)	90.9 (n=3)	88.4

What's quantized to what

Table with columns: tensors, format
tensors	format
all 40 layers' routed experts (`gate/up/down_proj`)	NVFP4 (group 16)
`self_attn` q/k/v/o, `linear_attn` in/out projections, shared-expert projections	FP8 E4M3
`lm_head`, embeddings, vision tower, MTP block, router gates, norms	BF16

Ornith-1.5-35B-A3B-agentic-NVFP4-FP8

README

Which of the two should you use

Serve it

Measured

What's quantized to what

Explore FriendliAI today

README

Which of the two should you use

Serve it

Measured

What's quantized to what