shisa-ai

Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill

What Changed

Base checkpoint: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
Donor MTP checkpoint: Qwen/Qwen3.6-35B-A3B
Added shard: model-mtp.safetensors
Added tensors: 19 top-level mtp.* tensors
MTP dtype: BF16
Base weights: unchanged FP8_BLOCK / compressed-tensors
Training: one KL-distill pass over a private local prompt/completion corpus, training only the MTP head while the Ornith verifier/base stayed frozen

The private training and benchmark prompts are not included in this repository because they were generated from personal rollouts and local repositories. The shareable recipe code is included under mtp_recipe/. The full local pipeline for this 5K run, including teacher generation and MTP KL training, completed within a few hours on a single RTX PRO 6000, so it should be practical to customize or improve for other workloads.

MTP Training

The KL-distill run trained only the MTP head. The Ornith base/verifier and shared lm_head stayed frozen.

Training setup:

Hardware: one NVIDIA RTX PRO 6000 Blackwell Workstation Edition
Trainable parameters: 844.6M MTP-head parameters
Base checkpoint: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
Initial MTP head: zero-training Qwen3.6 graft from Qwen/Qwen3.6-35B-A3B
Output checkpoint: this model
Objective: KL distribution matching against frozen Ornith verifier logits
Epochs: 1
Learning rate: 5e-5 with warmup ratio 0.05
:

Timing:

End-to-end distill command wall time: 71m 21s
Model load and checkpoint write overhead: about 34s
Approximate training-loop time: about 70-71m
Approximate optimizer-step time: 9.3s/step

Corpus/token counts:

Table with columns: Scope, Rows, Prompt/input tokens, Completion/output tokens, Notes
Scope	Rows	Prompt/input tokens	Completion/output tokens	Notes
Full teacher corpus	5,000	~28.62M	~3.66M	Raw generated private corpus
Train split raw	4,500	25.78M	3.29M	Before `max_seq_len` filtering
Distill processed	1,850	1.82M	1.11M in-window

The training split intentionally preserved long-context examples, but this run used max_seq_len=2048. As a result, 2,650/4,500 train rows were skipped because the prompt left fewer than four completion tokens inside the training window.

Approximate throughput over the full distill wall time:

Clipped forward tokens: ~2.92M tokens, or ~683 tok/s
KL-scored target tokens: ~1.108M tokens, or ~259 tok/s

One implementation detail from this run: the script processed 1,850 usable examples, but only 1,848 contributed to optimizer updates because the final two examples were left over after accumulation by four. This is negligible for this artifact, but a future distill run should flush the final partial accumulation step.

Important Result Summary

This checkpoint did improve the offline KL proxy, but it was not the best serving result in our matched vLLM tests. The best measured row was still the official zero-training Qwen3.6 MTP checkpoint with num_speculative_tokens=3.

Use shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP as the recommended throughput checkpoint from our testing. Use this distilled checkpoint as an experimental/diagnostic artifact unless your own serving tests show a win over that official MTP3 control.

Local Benchmark Methodology

Hardware:

One NVIDIA RTX PRO 6000 Blackwell Workstation Edition
Single-GPU serving on GPU0 only

Runtime:

vLLM 0.23.0
FlashInfer attention backend
FP8 KV cache
compressed-tensors quantization
No LMCache for the benchmark rows below
MAX_MODEL_LEN=32768
MAX_NUM_SEQS=16
MAX_BATCHED_TOKENS=32768
MAX_CUDAGRAPH_CAPTURE_SIZE=16
GPU_MEMORY_UTIL=0.95

Workload:

Private custom validation benchmark derived from local code/agentic prompts
64 requests
63,327 total input tokens
16,384 generated tokens (256 per request)
max_concurrency=1
request_rate=inf
temperature=0.6, top_p=0.95, top_k=20
ignore_eos

The private benchmark data is not uploaded. The benchmark used vLLM's bench serve --dataset-name custom path. To reproduce the command shape with your own non-private data, use a JSONL custom dataset with prompt and output_tokens fields and run a command like:

bash
CUDA_VISIBLE_DEVICES=0 vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions \
  --model ornith-35b-fp8-block-mtp-qwen36-distill \
  --tokenizer /path/to/Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill \
  --dataset-name custom \
  --dataset-path /path/to/custom-prompts.jsonl \
  --skip-chat-template \
  --disable-shuffle \
  --no-oversample \
  --num-prompts 64 \
  --custom-output-len -1 \
  --max-concurrency 1 \
  --request-rate inf \
  --temperature 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --ignore-eos

Local Results

Matched c=1 validation-prompt serving results:

Table with columns: Variant, MTP tokens, Output tok/s, Delta vs baseline, Median TTFT ms, Median TPOT ms, Median ITL ms, Acceptance
Variant	MTP tokens	Output tok/s	Delta vs baseline	Median TTFT ms	Median TPOT ms	Median ITL ms	Acceptance
Baseline no-spec	0	203.142	-	51.722	4.647	4.648	-
Official MTP, Qwen3.6 graft	1	221.032	+8.8%

Offline proxy on the same validation split:

Official MTP, Qwen3.6 graft post hidden: distribution-overlap acceptance proxy 0.841, mean KL 0.3162
Qwen3.6 KL-distill post hidden: distribution-overlap acceptance proxy 0.849, mean KL 0.2967

Interpretation: the KL pass improved the offline distribution metric, but did not improve matched vLLM serving throughput versus the official MTP3 control.

vLLM Usage

MTP serving requires a vLLM build that supports Qwen3.5 MoE MTP checkpoints. The local runs used vLLM 0.23.0.

bash
vllm serve /path/to/Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill \
  --served-model-name ornith-35b-fp8-block-mtp-qwen36-distill \
  --trust-remote-code \
  --quantization compressed-tensors \
  --language-model-only \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 32768 \
  --max-cudagraph-capture-size 16 \
  --attention-backend flashinfer \
  --kv-cache-dtype fp8 \
  --generation-config vllm \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

num_speculative_tokens=3 gave the best local throughput for this checkpoint, but it also reduced acceptance to about 67% and increased inter-token latency relative to no-spec serving. Re-benchmark on your hardware and workload before using it as a default.

Recipe Code

The non-private code used for grafting, corpus generation, KL distillation, offline proxy evaluation, and validation is included in mtp_recipe/. It includes a sanitized example config at mtp_recipe/configs/ornith-35b-distill-qwen36.example.yaml.

The private corpus and private benchmark JSONL used in the local run are not included.

Quantization Summary

The base checkpoint is unchanged from shisa-ai/Ornith-1.0-35B-FP8-BLOCK:

Source model: deepreinforce-ai/Ornith-1.0-35B
Quantization tool: llm-compressor model-free PTQ
Quantization format: compressed-tensors
Scheme: FP8_BLOCK
Calibration data: none; this is data-free/model-free PTQ
Weight quantization: static FP8, symmetric, block strategy, 128x128 blocks
Activation quantization: dynamic FP8, symmetric, group strategy, group size 128
Target modules: Linear
metadata version recorded in :

The quantization ignore list includes re:^mtp.*, so the MTP head remains BF16.

License and Attribution

The source Ornith model is MIT licensed. This derivative keeps the source license metadata and links to the upstream license file.

Attribution:

Source model: deepreinforce-ai/Ornith-1.0-35B
FP8_BLOCK base: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
MTP donor: Qwen/Qwen3.6-35B-A3B
MTP graft/distill recipe inspiration: protoLabsAI/Ornith-1.0-9B-MTP

If you use the source model, cite the original Ornith release:

bibtex
@misc{ornith-35b,
    title = {{Ornith-1.0-35B}: Agentic Coding, Open to All},
    url = {https://deep-reinforce.com/ornith_1_0.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

shisa-ai

Model Tree

Base

Qwen/Qwen3.6-35B-A3B

Quantized

this model

Input Modalities

Text

Image

Video

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

What Changed

Base checkpoint: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
Donor MTP checkpoint: Qwen/Qwen3.6-35B-A3B
Added shard: model-mtp.safetensors
Added tensors: 19 top-level mtp.* tensors
MTP dtype: BF16
Base weights: unchanged FP8_BLOCK / compressed-tensors
Training: one KL-distill pass over a private local prompt/completion corpus, training only the MTP head while the Ornith verifier/base stayed frozen

MTP Training

The KL-distill run trained only the MTP head. The Ornith base/verifier and shared lm_head stayed frozen.

Training setup:

Hardware: one NVIDIA RTX PRO 6000 Blackwell Workstation Edition
Trainable parameters: 844.6M MTP-head parameters
Base checkpoint: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
Initial MTP head: zero-training Qwen3.6 graft from Qwen/Qwen3.6-35B-A3B
Output checkpoint: this model
Objective: KL distribution matching against frozen Ornith verifier logits
Epochs: 1
Learning rate: 5e-5 with warmup ratio 0.05
:

Timing:

End-to-end distill command wall time: 71m 21s
Model load and checkpoint write overhead: about 34s
Approximate training-loop time: about 70-71m
Approximate optimizer-step time: 9.3s/step

Corpus/token counts:

Table with columns: Scope, Rows, Prompt/input tokens, Completion/output tokens, Notes
Scope	Rows	Prompt/input tokens	Completion/output tokens	Notes
Full teacher corpus	5,000	~28.62M	~3.66M	Raw generated private corpus
Train split raw	4,500	25.78M	3.29M	Before `max_seq_len` filtering
Distill processed	1,850	1.82M	1.11M in-window

Approximate throughput over the full distill wall time:

Clipped forward tokens: ~2.92M tokens, or ~683 tok/s
KL-scored target tokens: ~1.108M tokens, or ~259 tok/s

Important Result Summary

Local Benchmark Methodology

Hardware:

One NVIDIA RTX PRO 6000 Blackwell Workstation Edition
Single-GPU serving on GPU0 only

Runtime:

vLLM 0.23.0
FlashInfer attention backend
FP8 KV cache
compressed-tensors quantization
No LMCache for the benchmark rows below
MAX_MODEL_LEN=32768
MAX_NUM_SEQS=16
MAX_BATCHED_TOKENS=32768
MAX_CUDAGRAPH_CAPTURE_SIZE=16
GPU_MEMORY_UTIL=0.95

Workload:

Private custom validation benchmark derived from local code/agentic prompts
64 requests
63,327 total input tokens
16,384 generated tokens (256 per request)
max_concurrency=1
request_rate=inf
temperature=0.6, top_p=0.95, top_k=20
ignore_eos

bash
CUDA_VISIBLE_DEVICES=0 vllm bench serve \
  --backend vllm \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions \
  --model ornith-35b-fp8-block-mtp-qwen36-distill \
  --tokenizer /path/to/Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill \
  --dataset-name custom \
  --dataset-path /path/to/custom-prompts.jsonl \
  --skip-chat-template \
  --disable-shuffle \
  --no-oversample \
  --num-prompts 64 \
  --custom-output-len -1 \
  --max-concurrency 1 \
  --request-rate inf \
  --temperature 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --ignore-eos

Local Results

Matched c=1 validation-prompt serving results:

Table with columns: Variant, MTP tokens, Output tok/s, Delta vs baseline, Median TTFT ms, Median TPOT ms, Median ITL ms, Acceptance
Variant	MTP tokens	Output tok/s	Delta vs baseline	Median TTFT ms	Median TPOT ms	Median ITL ms	Acceptance
Baseline no-spec	0	203.142	-	51.722	4.647	4.648	-
Official MTP, Qwen3.6 graft	1	221.032	+8.8%

Offline proxy on the same validation split:

Official MTP, Qwen3.6 graft post hidden: distribution-overlap acceptance proxy 0.841, mean KL 0.3162
Qwen3.6 KL-distill post hidden: distribution-overlap acceptance proxy 0.849, mean KL 0.2967

Interpretation: the KL pass improved the offline distribution metric, but did not improve matched vLLM serving throughput versus the official MTP3 control.

vLLM Usage

MTP serving requires a vLLM build that supports Qwen3.5 MoE MTP checkpoints. The local runs used vLLM 0.23.0.

bash
vllm serve /path/to/Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill \
  --served-model-name ornith-35b-fp8-block-mtp-qwen36-distill \
  --trust-remote-code \
  --quantization compressed-tensors \
  --language-model-only \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 32768 \
  --max-cudagraph-capture-size 16 \
  --attention-backend flashinfer \
  --kv-cache-dtype fp8 \
  --generation-config vllm \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Recipe Code

The private corpus and private benchmark JSONL used in the local run are not included.

Quantization Summary

The base checkpoint is unchanged from shisa-ai/Ornith-1.0-35B-FP8-BLOCK:

Source model: deepreinforce-ai/Ornith-1.0-35B
Quantization tool: llm-compressor model-free PTQ
Quantization format: compressed-tensors
Scheme: FP8_BLOCK
Calibration data: none; this is data-free/model-free PTQ
Weight quantization: static FP8, symmetric, block strategy, 128x128 blocks
Activation quantization: dynamic FP8, symmetric, group strategy, group size 128
Target modules: Linear
metadata version recorded in :

The quantization ignore list includes re:^mtp.*, so the MTP head remains BF16.

License and Attribution

The source Ornith model is MIT licensed. This derivative keeps the source license metadata and links to the upstream license file.

Attribution:

Source model: deepreinforce-ai/Ornith-1.0-35B
FP8_BLOCK base: shisa-ai/Ornith-1.0-35B-FP8-BLOCK
MTP donor: Qwen/Qwen3.6-35B-A3B
MTP graft/distill recipe inspiration: protoLabsAI/Ornith-1.0-9B-MTP

If you use the source model, cite the original Ornith release:

bibtex
@misc{ornith-35b,
    title = {{Ornith-1.0-35B}: Agentic Coding, Open to All},
    url = {https://deep-reinforce.com/ornith_1_0.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}

Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill

README

What Changed

MTP Training

Important Result Summary

Local Benchmark Methodology

Local Results

vLLM Usage

Recipe Code

Quantization Summary

License and Attribution

Explore FriendliAI today

README

What Changed

MTP Training

Important Result Summary

Local Benchmark Methodology

Local Results

vLLM Usage

Recipe Code

Quantization Summary

License and Attribution