barryke

Ornith-1.0-9B-FP8-DYNAMIC

Why this quantization?

FP8_DYNAMIC is the simplest scheme that recovers near-full accuracy on most LLMs while cutting weight size roughly in half:

No calibration dataset required — activation scales are computed on the fly per token at inference time.
Per-channel weight scales avoid the accuracy loss that per-tensor weight quantization causes on outlier channels.
Runs at full speed on Ada Lovelace / Hopper / Blackwell GPUs (compute capability ≥ 8.9).

The vision tower and the hybrid Gated-DeltaNet attention projections are sensitive and left in BF16, matching the recipe used inside the quantization script. The lm_head (≈248k-token vocabulary) is also left in BF16.

Size on disk

Table with columns: Checkpoint, Weights on disk
Checkpoint	Weights on disk
Original (`deepreinforce-ai/Ornith-1.0-9B`, BF16)	18.820 GB
This repo (FP8-DYNAMIC)	13.520 GB
Reduction	28.2 % smaller

Quantization recipe

Reproduced verbatim from the Modal run that produced this checkpoint:

python
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=[
        "lm_head",             # 248k-token vocab, sensitive
        "re:.*visual.*",       # vision tower -> kept at BF16
        "re:.*linear_attn.*",  # hybrid Gated-DeltaNet projections
    ],
)
oneshot(model=model, recipe=recipe)

The pipeline (load → quantize → save → sanity-check generate) was orchestrated on a Modal H100 container. End-to-end cost for a 9B model is on the order of ~$0.30 per run.

Verification

Side-by-side generation against the original BF16 checkpoint on the same prompt (greedy decoding). Both checkpoints produced byte-identical output, confirming FP8-DYNAMIC recovery on this workload:

Prompt: The capital of Poland and Australia is

markdown
ORIGINAL  -> Thinking Process:

1.  **Analyze the Request:** The user is asking for the capital of Poland and Australia. This is a factual question with two parts.

2.  **Identify the Capitals:**
    *   Poland: Warsaw (Warszawa).
    *   Australia: Canberra.

3.  **Formulate the Answer:** Combine the two facts into a clear sentence.

4.  **Review for Accuracy:**
    *   Is Warsaw the capital of Poland? Yes.
    *   Is Canberra the capital of Australia? Yes.

5.  **Draft the Response:** "The capital of Poland is Warsaw, and the capital of Australia is Canberra."

6.  **Final Polish:** Keep it concise and direct. "The capital of Poland is Warsaw, and the capital of Australia is Canberra." or simply list them. A sentence is better
for flow.

7.  **Output Generation:** (Matches the drafted response)cw
</think>

The capital of Poland is **Warsaw**, and the capital of Australia is **Canberra**.

QUANTIZED -> Thinking Process:

1.  **Analyze the Request:** The user is asking for the capital of Poland and Australia. This is a factual question with two parts.

2.  **Identify the Capitals:**
    *   Poland: Warsaw (Warszawa).
    *   Australia: Canberra.

3.  **Formulate the Answer:** Combine the two facts into a clear sentence.

4.  **Review for Accuracy:**
    *   Is Warsaw the capital of Poland? Yes.
    *   Is Canberra the capital of Australia? Yes.

5.  **Draft the Response:** "The capital of Poland is Warsaw, and the capital of Australia is Canberra."

6.  **Final Polish:** Keep it concise and direct. "The capital of Poland is Warsaw, and the capital of Australia is Canberra." or simply list them. A sentence is better
for flow.

7.  **Output Generation:** (Matches the drafted response)cw
</think>

The capital of Poland is **Warsaw**, and the capital of Australia is **Canberra**.

Usage

vLLM (recommended for serving)

bash
vllm serve barryke/Ornith-1.0-9B-FP8-DYNAMIC \
    --served-model-name Ornith-1.0-9B \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 262144 \
    --gpu-memory-utilization 0.90 \
    --enable-prefix-caching \
    --enable-auto-tool-choice --tool-call-parser qwen3_xml \
    --reasoning-parser qwen3 \
    --trust-remote-code

The FP8 kernels are picked up automatically — no extra flags needed.

Hugging Face Transformers

Requires transformers >= 5.8.1 and compressed-tensors installed. The checkpoint loads via the same multimodal class the base model uses:

python
import torch
from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration

model_name = "barryke/Ornith-1.0-9B-FP8-DYNAMIC"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    model_name, dtype=torch.bfloat16, device_map="auto"
)

messages = [{"role": "user", "content": "Write a Python function is_prime(n). Keep it short."}]
templated = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(templated, return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=512, do_sample=True,
                     temperature=0.6, top_p=0.95, top_k=20)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

OpenAI-compatible API

Once vllm serve is running, any OpenAI SDK works:

python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Ornith-1.0-9B",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    temperature=0.6, top_p=0.95,
)
msg = response.choices[0].message
print("reasoning:", getattr(msg, "reasoning_content", None))
print("answer:", msg.content)

Hardware & latency notes

Inference: needs an FP8-capable GPU (Ada Lovelace / Hopper / Blackwell, i.e. compute capability ≥ 8.9). On older GPUs pick a W4A16 / INT8 quantization instead.
KV cache & context: unchanged from the base model (262k context window). The FP8 savings apply to weights only; KV cache stays in BF16/FP16.
Multimodal: the vision tower is kept at BF16, so image inputs still work. This checkpoint is text+image capable, just like the base.

Limitations

Same behavior as the base Ornith-1.0-9B (a reasoning model that emits <think>...</think> before its answer) — quantization introduces only negligible output drift, but evaluate on your own workload if exact reproduction matters.
Not calibrated against a specific downstream dataset; dynamic activations trade a small amount of peak throughput for zero-shot portability.

Citation

If you use this checkpoint, please cite the original Ornith model:

bibtex
@misc{ornith_9b,
    title = {{Ornith-1.0-9B}: Agentic Coding, Open to All},
    url = {https://deep-reinforce.com/ornith_1_0.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}

And reference the quantization toolchain:

bibtex
@misc{llmcompressor,
    title  = {LLM Compressor},
    author = {Neural Magic},
    url    = {https://github.com/vllm-project/llm-compressor}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

barryke

Model Tree

Base

deepreinforce-ai/Ornith-1.0-9B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Why this quantization?

FP8_DYNAMIC is the simplest scheme that recovers near-full accuracy on most LLMs while cutting weight size roughly in half:

No calibration dataset required — activation scales are computed on the fly per token at inference time.
Per-channel weight scales avoid the accuracy loss that per-tensor weight quantization causes on outlier channels.
Runs at full speed on Ada Lovelace / Hopper / Blackwell GPUs (compute capability ≥ 8.9).

Size on disk

Table with columns: Checkpoint, Weights on disk
Checkpoint	Weights on disk
Original (`deepreinforce-ai/Ornith-1.0-9B`, BF16)	18.820 GB
This repo (FP8-DYNAMIC)	13.520 GB
Reduction	28.2 % smaller

Quantization recipe

Reproduced verbatim from the Modal run that produced this checkpoint:

python
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=[
        "lm_head",             # 248k-token vocab, sensitive
        "re:.*visual.*",       # vision tower -> kept at BF16
        "re:.*linear_attn.*",  # hybrid Gated-DeltaNet projections
    ],
)
oneshot(model=model, recipe=recipe)

The pipeline (load → quantize → save → sanity-check generate) was orchestrated on a Modal H100 container. End-to-end cost for a 9B model is on the order of ~$0.30 per run.

Verification

Prompt: The capital of Poland and Australia is

markdown
ORIGINAL  -> Thinking Process:

1.  **Analyze the Request:** The user is asking for the capital of Poland and Australia. This is a factual question with two parts.

2.  **Identify the Capitals:**
    *   Poland: Warsaw (Warszawa).
    *   Australia: Canberra.

3.  **Formulate the Answer:** Combine the two facts into a clear sentence.

4.  **Review for Accuracy:**
    *   Is Warsaw the capital of Poland? Yes.
    *   Is Canberra the capital of Australia? Yes.

5.  **Draft the Response:** "The capital of Poland is Warsaw, and the capital of Australia is Canberra."

6.  **Final Polish:** Keep it concise and direct. "The capital of Poland is Warsaw, and the capital of Australia is Canberra." or simply list them. A sentence is better
for flow.

7.  **Output Generation:** (Matches the drafted response)cw
</think>

The capital of Poland is **Warsaw**, and the capital of Australia is **Canberra**.

QUANTIZED -> Thinking Process:

1.  **Analyze the Request:** The user is asking for the capital of Poland and Australia. This is a factual question with two parts.

2.  **Identify the Capitals:**
    *   Poland: Warsaw (Warszawa).
    *   Australia: Canberra.

3.  **Formulate the Answer:** Combine the two facts into a clear sentence.

4.  **Review for Accuracy:**
    *   Is Warsaw the capital of Poland? Yes.
    *   Is Canberra the capital of Australia? Yes.

5.  **Draft the Response:** "The capital of Poland is Warsaw, and the capital of Australia is Canberra."

6.  **Final Polish:** Keep it concise and direct. "The capital of Poland is Warsaw, and the capital of Australia is Canberra." or simply list them. A sentence is better
for flow.

7.  **Output Generation:** (Matches the drafted response)cw
</think>

The capital of Poland is **Warsaw**, and the capital of Australia is **Canberra**.

Usage

vLLM (recommended for serving)

bash
vllm serve barryke/Ornith-1.0-9B-FP8-DYNAMIC \
    --served-model-name Ornith-1.0-9B \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 262144 \
    --gpu-memory-utilization 0.90 \
    --enable-prefix-caching \
    --enable-auto-tool-choice --tool-call-parser qwen3_xml \
    --reasoning-parser qwen3 \
    --trust-remote-code

The FP8 kernels are picked up automatically — no extra flags needed.

Hugging Face Transformers

Requires transformers >= 5.8.1 and compressed-tensors installed. The checkpoint loads via the same multimodal class the base model uses:

python
import torch
from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration

model_name = "barryke/Ornith-1.0-9B-FP8-DYNAMIC"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    model_name, dtype=torch.bfloat16, device_map="auto"
)

messages = [{"role": "user", "content": "Write a Python function is_prime(n). Keep it short."}]
templated = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(templated, return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=512, do_sample=True,
                     temperature=0.6, top_p=0.95, top_k=20)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

OpenAI-compatible API

Once vllm serve is running, any OpenAI SDK works:

python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Ornith-1.0-9B",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    temperature=0.6, top_p=0.95,
)
msg = response.choices[0].message
print("reasoning:", getattr(msg, "reasoning_content", None))
print("answer:", msg.content)

Hardware & latency notes

Inference: needs an FP8-capable GPU (Ada Lovelace / Hopper / Blackwell, i.e. compute capability ≥ 8.9). On older GPUs pick a W4A16 / INT8 quantization instead.
KV cache & context: unchanged from the base model (262k context window). The FP8 savings apply to weights only; KV cache stays in BF16/FP16.
Multimodal: the vision tower is kept at BF16, so image inputs still work. This checkpoint is text+image capable, just like the base.

Limitations

Same behavior as the base Ornith-1.0-9B (a reasoning model that emits <think>...</think> before its answer) — quantization introduces only negligible output drift, but evaluate on your own workload if exact reproduction matters.
Not calibrated against a specific downstream dataset; dynamic activations trade a small amount of peak throughput for zero-shot portability.

Citation

If you use this checkpoint, please cite the original Ornith model:

bibtex
@misc{ornith_9b,
    title = {{Ornith-1.0-9B}: Agentic Coding, Open to All},
    url = {https://deep-reinforce.com/ornith_1_0.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}

And reference the quantization toolchain:

bibtex
@misc{llmcompressor,
    title  = {LLM Compressor},
    author = {Neural Magic},
    url    = {https://github.com/vllm-project/llm-compressor}
}

Ornith-1.0-9B-FP8-DYNAMIC

README

Why this quantization?

Size on disk

Quantization recipe

Verification

Usage

vLLM (recommended for serving)

Hugging Face Transformers

OpenAI-compatible API

Hardware & latency notes

Limitations

Citation

Explore FriendliAI today

README

Why this quantization?

Size on disk

Quantization recipe

Verification

Usage

vLLM (recommended for serving)

Hugging Face Transformers

OpenAI-compatible API

Hardware & latency notes

Limitations

Citation