pottokao

Ornith-1.0-35B-Heretic-NVFP4-MTP

Model specs

Table

Base architecture	Qwen3.5-MoE (`qwen3_5_moe`, `Qwen3_5MoeForConditionalGeneration`)
Params	~35B total / ~3B active (A3B)
MoE	256 experts, top-8 routing
Layers / hidden	40 / 2048
Context	262,144 (256K) native — ~128K KV practical on 32 GB VRAM
Vocab	248,320
Uncensoring	Heretic abliteration
Quantization	NVFP4 weights + FP8 attention + FP8 KV cache (mixed precision), via NVIDIA `modelopt` 0.45.0
Speculative decoding	4-layer NVFP4 MTP head, `num_speculative_tokens` (K) = 4
On-disk size	~23.8 GB (4 shards)

What changed in MTP v2

The draft (MTP) head was retrained from a v2 warm start:

Data: code-primary distillation set + agent traces (v1 was more general-purpose).
Training fix: corrected LR-schedule units.
Architecture: 4 NVFP4-quantized MTP layers; experts in NVFP4, the rest BF16.
vLLM's native spec_step_idx % mtp_num_hidden_layers routing means each speculative position uses its own dedicated layer (mtp_num_hidden_layers = 4), zero patching.

Offline validation (per-position acceptance, layers 0→3):

Table with columns: Head, pos0, pos1, pos2, pos3
Head	pos0	pos1	pos2	pos3
v1	0.7550	0.6658	0.6224	0.5901
v2 (this)	0.7638	0.6745	0.6332	0.6016

Performance — where the MTP head pays off

Measured single-stream on 2× RTX 5070 Ti (32 GB total, TP2), K=4, humming backend, via the repo's frozen probe (mtp_probe.py, which logs every run — reproducible with one command). The head is trained for agentic-coding / structured output, and that's where the speedup lands.

Agent / tool-calling (the trained target) — per-position acceptance at K=4:

Table with columns: draft position, 0, 1, 2, 3
draft position	0	1	2	3
acceptance	92%	83%	75%	67%

→ ~3.1 of 4 draft tokens accepted per step (~4 tokens generated per forward pass). Even the 4th position clears ~67%, so all four layers earn their slot — this is why K=4, not K=2/K=3. Single-stream ~370 tok/s, peaks ~405.

Compare accepted-tokens-per-step across configs, not aggregate acceptance % — a K=2 head only drafts the two easy leading positions, so its average % looks higher while producing fewer tokens per step.

Acceptance is content-dependent (open-ended generation jitters run-to-run; fixed-format is tight):

Table with columns: content, accepted tokens/step, single-stream tok/s
content	accepted tokens/step	single-stream tok/s
agent / tool-calling	~3.1 / 4	~370 (peak 405)
code	~2.7 / 4	~335
structured report	~2.2 / 4	~300
free prose	~1.1 / 4	~200

Runs entirely on two consumer 16 GB cards — no datacenter GPU needed.

Warm multi-turn (real agent-loop usage). The per-content tok/s above are single-shot (one cold request). In a sustained conversation — prefix cache warm, context growing — steady-state decode reaches ~440–450 tok/s across content (canon.py, 5-turn), the more representative figure for multi-turn agentic work.

What the MTP head was (and wasn't) trained for. The draft head is purpose-trained on agentic-coding — agent traces, tool-calling, and code. It was not trained on free prose, creative writing, or open-ended chat; on those the draft acceptance drops to ~30% and you get little speculative speedup (decode drops to ~200 tok/s). Note this affects speed only — the base Ornith model still answers prose/chat at full quality, you just don't get the MTP acceleration there.

Internal quality/speed A/B

⚠️ This is an internal 60-prompt A/B (code / hard-reasoning / mixed, 20 each), self-assessed and single-stream — it is not a standard public benchmark. Reported for transparency, not as a leaderboard claim.

Table with columns: Ornith 35B-A3B (this), A dense 27B ref
	Ornith 35B-A3B (this)	A dense 27B ref
Correct (self-judged)	59/60	60/60
Total time (60 prompts)	167 s	412 s
Avg speed (single-stream)	280.8 tok/s	114.6 tok/s

Takeaway: the A3B MoE decodes ~2.5× faster than a comparable dense model at near-equal answer quality, because only ~3B params are active per token.

Benchmark — ASUS GX10 (GB10), second platform

Measured on NVIDIA GB10 (unified 121 GiB LPDDR5X, single device, TP1), batch-1 single-stream, K=4, FP8 KV cache, humming MoE backend, vLLM nightly. Harness: Spec-Bench (gatekeeper) + held-out agent-code (main judge). Output is lossless (target-verified).

Compared on the same GB10 box against a DFlash baseline — AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4

z-lab/Qwen3.6-35B-A3B-DFlash (K=11, BF16 KV).

Table with columns: Category, MTP K=4 tok/s, MTP acc-len, MTP acc-rate, DFlash tok/s, DFlash acc-len, DFlash acc-rate
Category	MTP K=4 tok/s	MTP acc-len	MTP acc-rate	DFlash tok/s	DFlash acc-len	DFlash acc-rate
agent (held-out)	75.9	3.30	58%	64.0	4.97	36%
code (held-out)	65.4	2.86	47%	70.1

acc-rate = accepted / drafted per step ≈ (acc-len − 1) / K. The MTP head accepts ~58% of its 4 drafts vs ~32% for DFlash's 11 — nearly 2× the per-draft efficiency: it drafts a few high-confidence tokens instead of casting a wide net.
acc-len (tokens/step) favours DFlash's deeper draft, but on a single bandwidth-bound GB10 that depth converts to only ~6% average throughput.
Net: comparable single-stream speed at 1/3 the draft width and half the KV memory (FP8) — precision over volume. On the trained agent target, first-position acceptance stays 90%+.

Not a single-variable A/B — base model, speculative method, and KV dtype (FP8 vs BF16) all differ. GB10 is a single-device part; the 2× RTX 5070 Ti (TP2) numbers above are the high-bandwidth reference.

Benchmark — MTP vs DFlash, controlled 27B comparison (2× RTX 5070 Ti)

Ornith's 35B-A3B has no same-size public DFlash draft, so the cleanest MTP-vs-DFlash test is run on our 27B dense sibling (same base family, same rig, same harness). Hardware: 2× RTX 5070 Ti (16 GB each = 32 GB, TP2). All three use FP8 KV.

Table with columns: Category, MTP K=3 (128k ctx), MTP K=4 (96k ctx), DFlash K=15 (8k benched, ≤~32k)
Category	MTP K=3 (128k ctx)	MTP K=4 (96k ctx)	DFlash K=15 (8k benched, ≤~32k)
	tok/s · acc-len	tok/s · acc-len	tok/s · acc-len
agent (held-out)	64.6 · 3.34	62.1 · 3.79	60.5 · 4.83
code (held-out)	57.0 · 3.10	55.1 · 3.51	52.4 · 4.59
coding	109.1 · 3.26	102.9 · 3.65	111.1 · 5.29

Speed is a three-way tie (~85–89 tok/s). DFlash's deeper acceptance (acc-len 4.93 vs 3.2–3.6) does not convert to throughput — its 15-token draft cost cancels the gain.
MTP K=3 is fastest and leanest: a 3-token draft at 73% acceptance vs DFlash's 26%. Precision beats casting a wide net (acc-rate = accepted / drafted ≈ (acc-len − 1) / K).
Context is the real gap — MTP 96–128k vs DFlash 8k. DFlash needs a separate draft model that consumes VRAM, so on 2× 16 GB the context window stays small (~8k benched, ~32k practical ceiling). The MTP head is grafted into the target (near-zero extra VRAM), so it keeps the full 96–128k window.
Net on memory-constrained multi-GPU: MTP matches DFlash speed at 1/5 the draft width, ~3× the per-draft efficiency, and 3–16× the usable context (128k vs ~8–32k).

Why an external benchmark?

The MTP draft head is trained only on an agent-coding / tool-calling distribution. Crucially, this can affect exactly one thing: the draft acceptance rate — how often the head's guesses match the target, i.e. the speedup. It can never alter the model's output: speculative decoding is lossless, the target verifies every token, and answers are identical whether the head is accurate or not. Our concern was narrow but real — could specializing the head on agent-code severely degrade its acceptance rate on task types it was never trained on (translation, summarization, QA, math, RAG, coding)? To test exactly that, we add Spec-Bench, an external suite of untrained task types, as a regression gate on acceptance. It guards speed generalization only; output quality is unaffected by construction and out of scope.

Usage (vLLM)

Requires a vLLM build that supports qwen3_5_moe + MTP speculative decoding + NVFP4 MoE. NVFP4 MoE is not accepted by the triton backend — you must use the humming MoE backend.

bash
# 1. Download the weights from the Hub
hf download pottokao/Ornith-1.0-35B-Heretic-NVFP4-MTP --local-dir ./ornith-heretic

# 2. Serve with vLLM (2× 16 GB GPU, TP2)
docker run -d --name ornith --gpus all \
  -e NCCL_P2P_DISABLE=1 \
  -e VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE=134217728 \
  -v $(pwd)/ornith-heretic:/model \
  -p 8000:8000 \
  <your-vllm-image> \
    --host 0.0.0.0 --port 8000 \
    --model /model \
    --tensor-parallel-size 2 \
    --trust-remote-code \
    --kv-cache-dtype fp8 \
    --moe-backend humming \
    --attention-backend TRITON_ATTN \
    --gpu-memory-utilization 0.92 \
    --max-model-len 131072 \
    --max-num-seqs 12 \
    --max-num-batched-tokens 4096 \
    --enable-chunked-prefill --enable-prefix-caching --async-scheduling \
    --speculative-config '{"method":"mtp","num_speculative_tokens":4,"moe_backend":"humming","attention_backend":"TRITON_ATTN"}' \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_xml --enable-auto-tool-choice \
    --default-chat-template-kwargs '{"enable_thinking": false}'

Key flags (why each matters)

Table with columns: Flag, Why
Flag	Why
`--moe-backend humming`	Mandatory. NVFP4 MoE is rejected by the `triton` backend.
`--attention-backend TRITON_ATTN`	Unlocks FULL cudagraph with the MTP drafter.
`--speculative-config '{... "method":"mtp","num_speculative_tokens":4 ...}'`	Enables the 4-layer MTP head at K=4.
`--tool-call-parser qwen3_xml --enable-auto-tool-choice`	Needed for agentic tool use — this model is tuned for agent-code; without it you lose tool calling.
`--reasoning-parser qwen3`

Notes:

Recommended K=4. The 4-layer head is trained for it; running at K<4 wastes the extra layers, and the head is tuned for agentic-coding acceptance rates at K=4.
Memory / context: on 32 GB total VRAM (2× 16 GB, TP2) you can serve ~128K KV context (--max-model-len 131072). The base architecture supports 256K positions, but 128K is the practical KV budget at this VRAM after NVFP4 weights (~24 GB).
Tested TP2 on 2× RTX 5070 Ti (16 GB each). --max-num-seqs 12 is tuned for that VRAM; raise it if you have more headroom.

Limitations & responsible use

Uncensored: safety alignment has been removed (abliteration). This model will follow instructions the original refused. You are responsible for how you deploy it and for complying with applicable laws and the base model's license/usage policy.
Text-only: the vision/video capability of the base checkpoint is not available here.
Serving-specific: the MTP head needs the humming MoE backend and an NVFP4-aware vLLM; it will not run on stock/older engines without those.
Benchmarks above are self-run and single-stream; treat concurrency/throughput numbers as untested unless you measure them yourself.

Acknowledgements

Base model: deepreinforce-ai/Ornith-1.0-35B by DeepReinforce AI (MIT). Post-trained on Qwen3.5 (Alibaba Qwen team). All base-model credit is theirs; this repo is a downstream inference-optimized derivative.
Abliteration: Heretic by Philipp Emanuel Weidmann.
Quantization: NVIDIA TensorRT Model Optimizer (modelopt 0.45.0).
MTP head trained and grafted in-house.

License

MIT, inherited from the base model. Redistribution keeps the MIT notice and attribution to DeepReinforce AI.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

pottokao

Model Tree

Base

ornith-ai/Ornith-1.0-35B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Model specs

Table

Base architecture	Qwen3.5-MoE (`qwen3_5_moe`, `Qwen3_5MoeForConditionalGeneration`)
Params	~35B total / ~3B active (A3B)
MoE	256 experts, top-8 routing
Layers / hidden	40 / 2048
Context	262,144 (256K) native — ~128K KV practical on 32 GB VRAM
Vocab	248,320
Uncensoring	Heretic abliteration
Quantization	NVFP4 weights + FP8 attention + FP8 KV cache (mixed precision), via NVIDIA `modelopt` 0.45.0
Speculative decoding	4-layer NVFP4 MTP head, `num_speculative_tokens` (K) = 4
On-disk size	~23.8 GB (4 shards)

What changed in MTP v2

The draft (MTP) head was retrained from a v2 warm start:

Data: code-primary distillation set + agent traces (v1 was more general-purpose).
Training fix: corrected LR-schedule units.
Architecture: 4 NVFP4-quantized MTP layers; experts in NVFP4, the rest BF16.
vLLM's native spec_step_idx % mtp_num_hidden_layers routing means each speculative position uses its own dedicated layer (mtp_num_hidden_layers = 4), zero patching.

Offline validation (per-position acceptance, layers 0→3):

Table with columns: Head, pos0, pos1, pos2, pos3
Head	pos0	pos1	pos2	pos3
v1	0.7550	0.6658	0.6224	0.5901
v2 (this)	0.7638	0.6745	0.6332	0.6016

Performance — where the MTP head pays off

Agent / tool-calling (the trained target) — per-position acceptance at K=4:

Table with columns: draft position, 0, 1, 2, 3
draft position	0	1	2	3
acceptance	92%	83%	75%	67%

Compare accepted-tokens-per-step across configs, not aggregate acceptance % — a K=2 head only drafts the two easy leading positions, so its average % looks higher while producing fewer tokens per step.

Acceptance is content-dependent (open-ended generation jitters run-to-run; fixed-format is tight):

Table with columns: content, accepted tokens/step, single-stream tok/s
content	accepted tokens/step	single-stream tok/s
agent / tool-calling	~3.1 / 4	~370 (peak 405)
code	~2.7 / 4	~335
structured report	~2.2 / 4	~300
free prose	~1.1 / 4	~200

Runs entirely on two consumer 16 GB cards — no datacenter GPU needed.

Internal quality/speed A/B

⚠️ This is an internal 60-prompt A/B (code / hard-reasoning / mixed, 20 each), self-assessed and single-stream — it is not a standard public benchmark. Reported for transparency, not as a leaderboard claim.

Table with columns: Ornith 35B-A3B (this), A dense 27B ref
	Ornith 35B-A3B (this)	A dense 27B ref
Correct (self-judged)	59/60	60/60
Total time (60 prompts)	167 s	412 s
Avg speed (single-stream)	280.8 tok/s	114.6 tok/s

Takeaway: the A3B MoE decodes ~2.5× faster than a comparable dense model at near-equal answer quality, because only ~3B params are active per token.

Benchmark — ASUS GX10 (GB10), second platform

Compared on the same GB10 box against a DFlash baseline — AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4

z-lab/Qwen3.6-35B-A3B-DFlash (K=11, BF16 KV).

Table with columns: Category, MTP K=4 tok/s, MTP acc-len, MTP acc-rate, DFlash tok/s, DFlash acc-len, DFlash acc-rate
Category	MTP K=4 tok/s	MTP acc-len	MTP acc-rate	DFlash tok/s	DFlash acc-len	DFlash acc-rate
agent (held-out)	75.9	3.30	58%	64.0	4.97	36%
code (held-out)	65.4	2.86	47%	70.1

acc-rate = accepted / drafted per step ≈ (acc-len − 1) / K. The MTP head accepts ~58% of its 4 drafts vs ~32% for DFlash's 11 — nearly 2× the per-draft efficiency: it drafts a few high-confidence tokens instead of casting a wide net.
acc-len (tokens/step) favours DFlash's deeper draft, but on a single bandwidth-bound GB10 that depth converts to only ~6% average throughput.
Net: comparable single-stream speed at 1/3 the draft width and half the KV memory (FP8) — precision over volume. On the trained agent target, first-position acceptance stays 90%+.

Not a single-variable A/B — base model, speculative method, and KV dtype (FP8 vs BF16) all differ. GB10 is a single-device part; the 2× RTX 5070 Ti (TP2) numbers above are the high-bandwidth reference.

Benchmark — MTP vs DFlash, controlled 27B comparison (2× RTX 5070 Ti)

Table with columns: Category, MTP K=3 (128k ctx), MTP K=4 (96k ctx), DFlash K=15 (8k benched, ≤~32k)
Category	MTP K=3 (128k ctx)	MTP K=4 (96k ctx)	DFlash K=15 (8k benched, ≤~32k)
	tok/s · acc-len	tok/s · acc-len	tok/s · acc-len
agent (held-out)	64.6 · 3.34	62.1 · 3.79	60.5 · 4.83
code (held-out)	57.0 · 3.10	55.1 · 3.51	52.4 · 4.59
coding	109.1 · 3.26	102.9 · 3.65	111.1 · 5.29

Speed is a three-way tie (~85–89 tok/s). DFlash's deeper acceptance (acc-len 4.93 vs 3.2–3.6) does not convert to throughput — its 15-token draft cost cancels the gain.
MTP K=3 is fastest and leanest: a 3-token draft at 73% acceptance vs DFlash's 26%. Precision beats casting a wide net (acc-rate = accepted / drafted ≈ (acc-len − 1) / K).
Context is the real gap — MTP 96–128k vs DFlash 8k. DFlash needs a separate draft model that consumes VRAM, so on 2× 16 GB the context window stays small (~8k benched, ~32k practical ceiling). The MTP head is grafted into the target (near-zero extra VRAM), so it keeps the full 96–128k window.
Net on memory-constrained multi-GPU: MTP matches DFlash speed at 1/5 the draft width, ~3× the per-draft efficiency, and 3–16× the usable context (128k vs ~8–32k).

Why an external benchmark?

Usage (vLLM)

Requires a vLLM build that supports qwen3_5_moe + MTP speculative decoding + NVFP4 MoE. NVFP4 MoE is not accepted by the triton backend — you must use the humming MoE backend.

bash
# 1. Download the weights from the Hub
hf download pottokao/Ornith-1.0-35B-Heretic-NVFP4-MTP --local-dir ./ornith-heretic

# 2. Serve with vLLM (2× 16 GB GPU, TP2)
docker run -d --name ornith --gpus all \
  -e NCCL_P2P_DISABLE=1 \
  -e VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE=134217728 \
  -v $(pwd)/ornith-heretic:/model \
  -p 8000:8000 \
  <your-vllm-image> \
    --host 0.0.0.0 --port 8000 \
    --model /model \
    --tensor-parallel-size 2 \
    --trust-remote-code \
    --kv-cache-dtype fp8 \
    --moe-backend humming \
    --attention-backend TRITON_ATTN \
    --gpu-memory-utilization 0.92 \
    --max-model-len 131072 \
    --max-num-seqs 12 \
    --max-num-batched-tokens 4096 \
    --enable-chunked-prefill --enable-prefix-caching --async-scheduling \
    --speculative-config '{"method":"mtp","num_speculative_tokens":4,"moe_backend":"humming","attention_backend":"TRITON_ATTN"}' \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_xml --enable-auto-tool-choice \
    --default-chat-template-kwargs '{"enable_thinking": false}'

Key flags (why each matters)

Table with columns: Flag, Why
Flag	Why
`--moe-backend humming`	Mandatory. NVFP4 MoE is rejected by the `triton` backend.
`--attention-backend TRITON_ATTN`	Unlocks FULL cudagraph with the MTP drafter.
`--speculative-config '{... "method":"mtp","num_speculative_tokens":4 ...}'`	Enables the 4-layer MTP head at K=4.
`--tool-call-parser qwen3_xml --enable-auto-tool-choice`	Needed for agentic tool use — this model is tuned for agent-code; without it you lose tool calling.
`--reasoning-parser qwen3`

Notes:

Recommended K=4. The 4-layer head is trained for it; running at K<4 wastes the extra layers, and the head is tuned for agentic-coding acceptance rates at K=4.
Memory / context: on 32 GB total VRAM (2× 16 GB, TP2) you can serve ~128K KV context (--max-model-len 131072). The base architecture supports 256K positions, but 128K is the practical KV budget at this VRAM after NVFP4 weights (~24 GB).
Tested TP2 on 2× RTX 5070 Ti (16 GB each). --max-num-seqs 12 is tuned for that VRAM; raise it if you have more headroom.

Limitations & responsible use

Uncensored: safety alignment has been removed (abliteration). This model will follow instructions the original refused. You are responsible for how you deploy it and for complying with applicable laws and the base model's license/usage policy.
Text-only: the vision/video capability of the base checkpoint is not available here.
Serving-specific: the MTP head needs the humming MoE backend and an NVFP4-aware vLLM; it will not run on stock/older engines without those.
Benchmarks above are self-run and single-stream; treat concurrency/throughput numbers as untested unless you measure them yourself.

Acknowledgements

Base model: deepreinforce-ai/Ornith-1.0-35B by DeepReinforce AI (MIT). Post-trained on Qwen3.5 (Alibaba Qwen team). All base-model credit is theirs; this repo is a downstream inference-optimized derivative.
Abliteration: Heretic by Philipp Emanuel Weidmann.
Quantization: NVIDIA TensorRT Model Optimizer (modelopt 0.45.0).
MTP head trained and grafted in-house.

License

MIT, inherited from the base model. Redistribution keeps the MIT notice and attribution to DeepReinforce AI.

Ornith-1.0-35B-Heretic-NVFP4-MTP

README

Model specs

What changed in MTP v2

Performance — where the MTP head pays off

Internal quality/speed A/B

Benchmark — ASUS GX10 (GB10), second platform

Benchmark — MTP vs DFlash, controlled 27B comparison (2× RTX 5070 Ti)

Why an external benchmark?

Usage (vLLM)

Key flags (why each matters)

Limitations & responsible use

Acknowledgements

License

Explore FriendliAI today

README

Model specs

What changed in MTP v2

Performance — where the MTP head pays off

Internal quality/speed A/B

Benchmark — ASUS GX10 (GB10), second platform

Benchmark — MTP vs DFlash, controlled 27B comparison (2× RTX 5070 Ti)

Why an external benchmark?

Usage (vLLM)

Key flags (why each matters)

Limitations & responsible use

Acknowledgements

License