shisa-ai

Ornith-1.0-35B-FP8-BLOCK-MTP

What Changed

Base checkpoint: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
Donor MTP checkpoint: Qwen/Qwen3.6-35B-A3B
Added shard: model-mtp.safetensors
Added tensors: 19 top-level mtp.* tensors
MTP dtype: BF16
Base weights: unchanged FP8_BLOCK / compressed-tensors
Training: none; this is a direct MTP tensor graft

No private training corpus is needed for this checkpoint because it is not distilled. The graft copies the donor mtp.* tensors into the Ornith FP8_BLOCK checkpoint and updates the safetensors index.

Important Result Summary

This is the strongest Ornith 35B MTP checkpoint from our local serving tests. It also beat the one-epoch Qwen3.6 KL-distilled derivative in matched vLLM serving, even though that distilled derivative had a slightly better offline KL proxy.

Recommended starting point:

Use num_speculative_tokens=3 when optimizing for output throughput.
Use num_speculative_tokens=1 or no speculative decoding if your workload is more sensitive to inter-token latency or acceptance stability.
Re-benchmark on your own traffic before making it a production default.

Local Benchmark Methodology

Hardware:

One NVIDIA RTX PRO 6000 Blackwell Workstation Edition
Single-GPU serving on GPU0 only

Runtime:

vLLM 0.23.0
FlashInfer attention backend
FP8 KV cache
compressed-tensors quantization
No LMCache for the benchmark rows below
MAX_MODEL_LEN=32768
MAX_NUM_SEQS=16
MAX_BATCHED_TOKENS=32768
MAX_CUDAGRAPH_CAPTURE_SIZE=16
GPU_MEMORY_UTIL=0.95

Workload:

Private custom validation benchmark derived from local code/agentic prompts
64 requests
63,327 total input tokens
16,384 generated tokens (256 per request)
max_concurrency=1
request_rate=inf
temperature=0.6, top_p=0.95, top_k=20
ignore_eos

The private benchmark data is not uploaded. The benchmark used vLLM's bench serve --dataset-name custom path. To reproduce the command shape with your own non-private data, use a JSONL custom dataset with prompt and output_tokens fields and run a command like:

bash
CUDA_VISIBLE_DEVICES=0 vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions \
  --model ornith-35b-fp8-block-mtp \
  --tokenizer shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP \
  --dataset-name custom \
  --dataset-path /path/to/custom-prompts.jsonl \
  --skip-chat-template \
  --disable-shuffle \
  --no-oversample \
  --num-prompts 64 \
  --custom-output-len -1 \
  --max-concurrency 1 \
  --request-rate inf \
  --temperature 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --ignore-eos

Local Results

Matched c=1 validation-prompt serving results:

Table with columns: Variant, MTP tokens, Output tok/s, Delta vs baseline, Median TTFT ms, Median TPOT ms, Median ITL ms, Acceptance
Variant	MTP tokens	Output tok/s	Delta vs baseline	Median TTFT ms	Median TPOT ms	Median ITL ms	Acceptance
Baseline no-spec	0	203.142	-	51.722	4.647	4.648	-
Official MTP, Qwen3.6 graft	1	221.032	+8.8%

Comparison against the companion KL-distilled artifact:

Table with columns: Variant, MTP tokens, Output tok/s, Delta vs baseline, Median TTFT ms, Median TPOT ms, Median ITL ms, Acceptance
Variant	MTP tokens	Output tok/s	Delta vs baseline	Median TTFT ms	Median TPOT ms	Median ITL ms	Acceptance
Official MTP, Qwen3.6 graft	3	246.522	+21.4%	65.482	3.571	10.739	66.98%
Qwen3.6 KL-distill	3	237.581	+17.0%

The distill checkpoint is planned as shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill. It improved the offline proxy, but it did not beat this official MTP checkpoint in matched serving throughput.

Offline proxy on the same validation split:

Official MTP, Qwen3.6 graft: distribution-overlap acceptance proxy 0.841, mean KL 0.3162
Qwen3.6 KL-distill: distribution-overlap acceptance proxy 0.849, mean KL 0.2967

Internal Alternatives Tested

We also tested a Qwen3.5 donor graft, but it is not planned for upload. The Qwen3.5 rows were from an earlier MTP1-only ShareGPT sweep, so they are useful as donor-selection context rather than a direct replacement for the validation table above.

Table with columns: Variant, c, Output tok/s, Delta vs baseline, Median TPOT ms, Acceptance
Variant	c	Output tok/s	Delta vs baseline	Median TPOT ms	Acceptance
Baseline no-spec	1	200.340	-	4.733	-
Qwen3.6 MTP1 graft	1	211.736	+5.7%	4.296	80.63%
Qwen3.5 MTP1 graft	1

The Qwen3.5 graft was marginally faster in those MTP1 rows, but it had lower acceptance. The later Qwen3.6 validation sweep at MTP3 produced the best retained throughput result, so Qwen3.6 is the donor used for the official upload.

vLLM Usage

MTP serving requires a vLLM build that supports Qwen3.5 MoE MTP checkpoints. The local runs used vLLM 0.23.0.

bash
vllm serve shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP \
  --served-model-name ornith-35b-fp8-block-mtp \
  --trust-remote-code \
  --quantization compressed-tensors \
  --language-model-only \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 32768 \
  --max-cudagraph-capture-size 16 \
  --attention-backend flashinfer \
  --kv-cache-dtype fp8 \
  --generation-config vllm \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

num_speculative_tokens=3 gave the best local output throughput for this checkpoint, but it also reduced acceptance to about 67% and increased inter-token latency relative to no-spec serving. Re-benchmark on your hardware and workload before using it as a default.

Quantization Summary

The base checkpoint is unchanged from shisa-ai/Ornith-1.0-35B-FP8-BLOCK:

Source model: deepreinforce-ai/Ornith-1.0-35B
Quantization tool: llm-compressor model-free PTQ
Quantization format: compressed-tensors
Scheme: FP8_BLOCK
Calibration data: none; this is data-free/model-free PTQ
Weight quantization: static FP8, symmetric, block strategy, 128x128 blocks
Activation quantization: dynamic FP8, symmetric, group strategy, group size 128
Target modules: Linear
metadata version recorded in :

The quantization ignore list includes re:^mtp.*, so the grafted MTP head remains BF16.

License and Attribution

The source Ornith model is MIT licensed. This derivative keeps the source license metadata and links to the upstream license file.

Attribution:

Source model: deepreinforce-ai/Ornith-1.0-35B
FP8_BLOCK base: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
MTP donor: Qwen/Qwen3.6-35B-A3B
MTP graft/distill recipe inspiration: protoLabsAI/Ornith-1.0-9B-MTP

If you use the source model, cite the original Ornith release:

bibtex
@misc{ornith-35b,
    title = {{Ornith-1.0-35B}: Agentic Coding, Open to All},
    url = {https://deep-reinforce.com/ornith_1_0.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

shisa-ai

Model Tree

Base

Qwen/Qwen3.6-35B-A3B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

What Changed

Base checkpoint: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
Donor MTP checkpoint: Qwen/Qwen3.6-35B-A3B
Added shard: model-mtp.safetensors
Added tensors: 19 top-level mtp.* tensors
MTP dtype: BF16
Base weights: unchanged FP8_BLOCK / compressed-tensors
Training: none; this is a direct MTP tensor graft

Important Result Summary

Recommended starting point:

Use num_speculative_tokens=3 when optimizing for output throughput.
Use num_speculative_tokens=1 or no speculative decoding if your workload is more sensitive to inter-token latency or acceptance stability.
Re-benchmark on your own traffic before making it a production default.

Local Benchmark Methodology

Hardware:

One NVIDIA RTX PRO 6000 Blackwell Workstation Edition
Single-GPU serving on GPU0 only

Runtime:

vLLM 0.23.0
FlashInfer attention backend
FP8 KV cache
compressed-tensors quantization
No LMCache for the benchmark rows below
MAX_MODEL_LEN=32768
MAX_NUM_SEQS=16
MAX_BATCHED_TOKENS=32768
MAX_CUDAGRAPH_CAPTURE_SIZE=16
GPU_MEMORY_UTIL=0.95

Workload:

Private custom validation benchmark derived from local code/agentic prompts
64 requests
63,327 total input tokens
16,384 generated tokens (256 per request)
max_concurrency=1
request_rate=inf
temperature=0.6, top_p=0.95, top_k=20
ignore_eos

bash
CUDA_VISIBLE_DEVICES=0 vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions \
  --model ornith-35b-fp8-block-mtp \
  --tokenizer shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP \
  --dataset-name custom \
  --dataset-path /path/to/custom-prompts.jsonl \
  --skip-chat-template \
  --disable-shuffle \
  --no-oversample \
  --num-prompts 64 \
  --custom-output-len -1 \
  --max-concurrency 1 \
  --request-rate inf \
  --temperature 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --ignore-eos

Local Results

Matched c=1 validation-prompt serving results:

Table with columns: Variant, MTP tokens, Output tok/s, Delta vs baseline, Median TTFT ms, Median TPOT ms, Median ITL ms, Acceptance
Variant	MTP tokens	Output tok/s	Delta vs baseline	Median TTFT ms	Median TPOT ms	Median ITL ms	Acceptance
Baseline no-spec	0	203.142	-	51.722	4.647	4.648	-
Official MTP, Qwen3.6 graft	1	221.032	+8.8%

Comparison against the companion KL-distilled artifact:

Table with columns: Variant, MTP tokens, Output tok/s, Delta vs baseline, Median TTFT ms, Median TPOT ms, Median ITL ms, Acceptance
Variant	MTP tokens	Output tok/s	Delta vs baseline	Median TTFT ms	Median TPOT ms	Median ITL ms	Acceptance
Official MTP, Qwen3.6 graft	3	246.522	+21.4%	65.482	3.571	10.739	66.98%
Qwen3.6 KL-distill	3	237.581	+17.0%

Offline proxy on the same validation split:

Official MTP, Qwen3.6 graft: distribution-overlap acceptance proxy 0.841, mean KL 0.3162
Qwen3.6 KL-distill: distribution-overlap acceptance proxy 0.849, mean KL 0.2967

Internal Alternatives Tested

Table with columns: Variant, c, Output tok/s, Delta vs baseline, Median TPOT ms, Acceptance
Variant	c	Output tok/s	Delta vs baseline	Median TPOT ms	Acceptance
Baseline no-spec	1	200.340	-	4.733	-
Qwen3.6 MTP1 graft	1	211.736	+5.7%	4.296	80.63%
Qwen3.5 MTP1 graft	1

vLLM Usage

MTP serving requires a vLLM build that supports Qwen3.5 MoE MTP checkpoints. The local runs used vLLM 0.23.0.

bash
vllm serve shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP \
  --served-model-name ornith-35b-fp8-block-mtp \
  --trust-remote-code \
  --quantization compressed-tensors \
  --language-model-only \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 32768 \
  --max-cudagraph-capture-size 16 \
  --attention-backend flashinfer \
  --kv-cache-dtype fp8 \
  --generation-config vllm \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Quantization Summary

The base checkpoint is unchanged from shisa-ai/Ornith-1.0-35B-FP8-BLOCK:

Source model: deepreinforce-ai/Ornith-1.0-35B
Quantization tool: llm-compressor model-free PTQ
Quantization format: compressed-tensors
Scheme: FP8_BLOCK
Calibration data: none; this is data-free/model-free PTQ
Weight quantization: static FP8, symmetric, block strategy, 128x128 blocks
Activation quantization: dynamic FP8, symmetric, group strategy, group size 128
Target modules: Linear
metadata version recorded in :

The quantization ignore list includes re:^mtp.*, so the grafted MTP head remains BF16.

License and Attribution

The source Ornith model is MIT licensed. This derivative keeps the source license metadata and links to the upstream license file.

Attribution:

Source model: deepreinforce-ai/Ornith-1.0-35B
FP8_BLOCK base: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
MTP donor: Qwen/Qwen3.6-35B-A3B
MTP graft/distill recipe inspiration: protoLabsAI/Ornith-1.0-9B-MTP

If you use the source model, cite the original Ornith release:

bibtex
@misc{ornith-35b,
    title = {{Ornith-1.0-35B}: Agentic Coding, Open to All},
    url = {https://deep-reinforce.com/ornith_1_0.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}

Ornith-1.0-35B-FP8-BLOCK-MTP

README

What Changed

Important Result Summary

Local Benchmark Methodology

Local Results

Internal Alternatives Tested

vLLM Usage

Quantization Summary

License and Attribution

Explore FriendliAI today

README

What Changed

Important Result Summary

Local Benchmark Methodology

Local Results

Internal Alternatives Tested

vLLM Usage

Quantization Summary

License and Attribution