programmer-666

Ornith-1.0-35B-oQ8e

Why this exists

The original Ornith-1.0-35B is released in BF16, which is heavy to load and run on consumer Apple hardware. This repo re-packages the model with oQe so that:

Peak memory usage drops substantially (see benchmarks below), making it practical on machines with less unified memory.
Decode (token generation) throughput improves, since generation is memory-bandwidth bound and benefits from the smaller weight footprint.
Quality is protected where it matters most: sensitive layers, MoE routers, shared experts, and vision/audio components are kept at higher precision instead of being blindly quantized (see below).
The model keeps its full 35B-parameter (approximately 3B active) MoE architecture, tool-calling ability, and reasoning (<think>) behavior intact; only the numeric representation of the weights changes.

About the base model: Ornith-1.0-35B

Ornith-1.0-35B is the lightweight member of DeepReinforce AI's Ornith-1.0 family, a self-improving set of open-source models specialized for agentic coding. It is post-trained on top of Qwen3.5 using reinforcement learning that jointly optimizes both the solution rollouts and the scaffolding that drives them.

Key architecture facts (inherited unchanged in this quantized build):

35B total parameters, approximately 3B activated per token (MoE)
256 experts (8 active per token + 1 shared expert)
Hybrid full-attention + linear (Gated DeltaNet) attention
Vision encoder included (image-text-to-text / multimodal)
Reasoning model: emits a <think>...</think> block before the final answer
Strong tool-calling / OpenAI-compatible function-calling support
State-of-the-art results among open models of comparable size on Terminal-Bench 2.1, SWE-bench Verified/Pro/Multilingual, and NL2Repo

For full details on training, benchmarks, and intended use, see the original model card.

Quantization method: oMLX Universal Dynamic Quantization

Quantization here isn't tied to any one inference server, it produces a standard mlx-lm model that works everywhere MLX safetensors are supported.

The quantizer streams tensors directly from the source safetensors, measures per-layer sensitivity, and writes a byte-budgeted mixed-precision checkpoint. There are two variants, oQ and oQe; this repo (oQ8e) uses oQe targeting an 8-bit average budget.

What standard oQ does:

Measures layer sensitivity using calibration prompts, comparing each quantized layer's output error against the float baseline.
Allocates higher bits to sensitive layers and protected tensors via mixed precision, while keeping the overall bits-per-weight budget under a hard cap.
Applies architecture-aware rules: MoE routers stay in fp16, shared experts and output-critical tensors are protected, vision/audio encoders are left unquantized, and SSM state tensors are preserved.

What oQe adds (this build):

Keeps the same oQ sensitivity plan, then layers an imatrix-weighted affine quantization step on top, the same idea llama.cpp's imatrix popularized: collect activation energy from representative prompts, then spend less quantization error on the input channels that matter most.
Concretely: oQe runs calibration samples through the model and records average squared input activation, E[x^2], per quantized Linear input channel.
For MoE SwitchLinear layers, it tracks one importance vector per expert, using router-selected expert indices, so rarely-activated experts still get proper coverage.
Calibration sample count adapts upward from the requested amount when MoE expert coverage isn't yet sufficient.
During affine quantization, scale/bias candidates are chosen to minimize sum(importance * (weight - dequantized_weight)^2) rather than plain unweighted MSE.
If a tensor has no matching imatrix entry, it falls back to standard oQ for that tensor (unless strict coverage is enabled, in which case conversion fails instead of silently falling back).

Table with columns: oQ, oQe (this repo)
	oQ	oQe (this repo)
Bit allocation	Layer-sensitivity mixed precision	Same oQ plan
Affine quantization	Standard min/max affine per group	imatrix-weighted clipping/search per group
Best for	Fast, deterministic conversion with strong baseline quality	Better low-bit retention, especially for MoE and activation-skewed layers

No fine-tuning, distillation, or weight editing was performed beyond this quantization pass; the underlying architecture, tokenizer, and chat template are identical to the base model.

Usage

This is a plain mlx-lm-format checkpoint, so any MLX-based tool works out of the box, no custom loader needed:

bash
pip install mlx-lm

python -m mlx_lm.generate \
  --model programmer-666/Ornith-1.0-35B-oQ8e \
  --prompt "Write a Python function is_prime(n). Keep it short." \
  --max-tokens 512

python
from mlx_lm import load, generate

model, tokenizer = load("programmer-666/Ornith-1.0-35B-oQ8e")
response = generate(model, tokenizer, prompt="Hello", max_tokens=256, verbose=True)
print(response)

For multimodal (image + text) use, the same checkpoint can be loaded with mlx-vlm since the vision encoder is preserved unquantized:

python
from mlx_vlm import load, generate

model, processor = load("programmer-666/Ornith-1.0-35B-oQ8e")
output = generate(model, processor, prompt="Hello", max_tokens=256, temp=1.0, top_p=1.0)
print(output)

It also works directly in LM Studio or oMLX, just search for or point either app at programmer-666/Ornith-1.0-35B-oQ8e.

Ornith is a reasoning model: the response will contain a <think>...</think> block before the final answer. Split on </think> if you only want the final answer.

Benchmarks

All figures below compare the BF16 BASE model against this oQ8e (oQe, 8-bit budget) build on identical prompt/generation configurations (pp<N>/tg128 = N prompt tokens, 128 generated tokens). (Test hardware: update this line with your specific Apple Silicon device and unified memory size.)

Single-request performance vs. context length

BASE (BF16)

Table with columns: Test, TTFT (ms), TPOT (ms), PP (tok/s), TG (tok/s), E2E (s), Throughput (tok/s), Peak Mem (GB)
Test	TTFT (ms)	TPOT (ms)	PP (tok/s)	TG (tok/s)	E2E (s)	Throughput (tok/s)	Peak Mem (GB)
pp1024/tg128	637.0	16.89	1607.4	59.7	2.791	412.8	65.62
pp4096/tg128	2187.3	17.24	1872.6

oQ8e (8-bit)

Table with columns: Test, TTFT (ms), TPOT (ms), PP (tok/s), TG (tok/s), E2E (s), Throughput (tok/s), Peak Mem (GB)
Test	TTFT (ms)	TPOT (ms)	PP (tok/s)	TG (tok/s)	E2E (s)	Throughput (tok/s)	Peak Mem (GB)
pp1024/tg128	603.0	12.34	1698.2	81.7	2.177	529.1	35.38
pp4096/tg128	2262.6	12.60	1810.3

Takeaways:

Memory: oQ8e cuts peak memory by roughly 43 to 46% across all context lengths (e.g. 65.62 GB to 35.38 GB at 1K context; 70.63 GB to 40.18 GB at 200K context).
Generation (TG) speed: oQ8e is about 31 to 37% faster at short-to-mid context (1K to 32K tokens), since decoding is memory-bandwidth bound and benefits directly from the smaller weights. The gap narrows around 64K context and becomes less consistent beyond that.
Prompt processing (PP) speed: Mixed results, oQ8e is slightly faster at very short prompts but 6 to 26% slower at mid-to-large prompts (8K to 131K tokens), which is the expected trade-off of 8-bit compute (dequantization overhead during compute-bound prefill).
Very long context (64K and above): Both TPOT and TG/PP throughput become noticeably less smooth for both models, expected behavior as unified memory pressure and cache effects start to dominate. This range is worth re-validating on your own hardware/config before relying on it.

Batching performance (fixed request size, varying concurrent batch)

BASE (BF16)

Table with columns: Batch, TG (tok/s), Speedup, PP (tok/s), PP per-request (tok/s), TTFT (ms), E2E (s)
Batch	TG (tok/s)	Speedup	PP (tok/s)	PP per-request (tok/s)	TTFT (ms)	E2E (s)
1	59.7	1.00	1607.4	1607.4	637.0	2.791
2	38.1	0.64	506.9	253.4	4040.3

oQ8e (8-bit)

Table with columns: Batch, TG (tok/s), Speedup, PP (tok/s), PP per-request (tok/s), TTFT (ms), E2E (s)
Batch	TG (tok/s)	Speedup	PP (tok/s)	PP per-request (tok/s)	TTFT (ms)	E2E (s)
1	81.7	1.00	1698.2	1698.2	603.0	2.177
2	128.6	1.57	269.5	253.9	3799.6

Takeaways:

oQ8e scales much better with concurrency: at batch size 8 it reaches a 2.45x speedup over its own batch-1 throughput, versus only 1.13x for the BF16 base model. The extra memory headroom freed by 8-bit weights lets more requests be batched before hitting memory/bandwidth limits.
End-to-end latency at batch 8 is roughly 1.8x lower with oQ8e (13.677 s vs 24.681 s), despite doing the same amount of work.
The BASE model actually regresses at batch size 2 (0.64x, worse than single-request), while oQ8e scales up from batch 1 immediately. This suggests the base BF16 model is memory-constrained enough that low-batch concurrency isn't worthwhile on this hardware, whereas oQ8e has enough headroom to benefit from batching right away.

Limitations

Quantization still introduces some numerical approximation vs. the original BF16 weights, even with oQe's sensitivity- and imatrix-aware protections. No formal accuracy/quality regression testing (e.g. on Terminal-Bench 2.1 or SWE-bench) has been run for this specific build; treat throughput/memory numbers above as validated, but re-check task quality for your own use case before production use.
Any tensor without a matching imatrix calibration entry silently falls back to standard oQ precision for that tensor, which is expected but worth knowing if you inspect per-tensor bit allocation.
Prompt-processing (prefill) throughput can be lower than the base model at large context sizes; this build is best suited to workloads dominated by generation rather than very long, one-off prompts.
Behavior beyond approximately 64K context tokens is less predictable for both the base and quantized model and should be validated on your target hardware.

License

Released under the MIT License, inherited from the base model deepreinforce-ai/Ornith-1.0-35B.

Citation

If you use this model, please cite the original Ornith-1.0-35B work:

bibtex
@misc{ornith-35b,
    title = {{Ornith-1.0-35B}: Agentic Coding, Open to All},
    url = {https://deep-reinforce.com/ornith_1_0.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}

And, if useful, this quantized build:

bibtex
@misc{ornith-35b-oq8e,
    title = {{Ornith-1.0-35B-oQ8e}: oMLX Universal Dynamic Quantization (oQe, 8-bit) for Apple Silicon},
    url = {https://huggingface.co/programmer-666/Ornith-1.0-35B-oQ8e},
    author = {{programmer-666}},
    year = {2026}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

programmer-666

Model Tree

Base

deepreinforce-ai/Ornith-1.0-35B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Why this exists

The original Ornith-1.0-35B is released in BF16, which is heavy to load and run on consumer Apple hardware. This repo re-packages the model with oQe so that:

Peak memory usage drops substantially (see benchmarks below), making it practical on machines with less unified memory.
Decode (token generation) throughput improves, since generation is memory-bandwidth bound and benefits from the smaller weight footprint.
Quality is protected where it matters most: sensitive layers, MoE routers, shared experts, and vision/audio components are kept at higher precision instead of being blindly quantized (see below).
The model keeps its full 35B-parameter (approximately 3B active) MoE architecture, tool-calling ability, and reasoning (<think>) behavior intact; only the numeric representation of the weights changes.

About the base model: Ornith-1.0-35B

Key architecture facts (inherited unchanged in this quantized build):

35B total parameters, approximately 3B activated per token (MoE)
256 experts (8 active per token + 1 shared expert)
Hybrid full-attention + linear (Gated DeltaNet) attention
Vision encoder included (image-text-to-text / multimodal)
Reasoning model: emits a <think>...</think> block before the final answer
Strong tool-calling / OpenAI-compatible function-calling support
State-of-the-art results among open models of comparable size on Terminal-Bench 2.1, SWE-bench Verified/Pro/Multilingual, and NL2Repo

For full details on training, benchmarks, and intended use, see the original model card.

Quantization method: oMLX Universal Dynamic Quantization

Quantization here isn't tied to any one inference server, it produces a standard mlx-lm model that works everywhere MLX safetensors are supported.

What standard oQ does:

Measures layer sensitivity using calibration prompts, comparing each quantized layer's output error against the float baseline.
Allocates higher bits to sensitive layers and protected tensors via mixed precision, while keeping the overall bits-per-weight budget under a hard cap.
Applies architecture-aware rules: MoE routers stay in fp16, shared experts and output-critical tensors are protected, vision/audio encoders are left unquantized, and SSM state tensors are preserved.

What oQe adds (this build):

Keeps the same oQ sensitivity plan, then layers an imatrix-weighted affine quantization step on top, the same idea llama.cpp's imatrix popularized: collect activation energy from representative prompts, then spend less quantization error on the input channels that matter most.
Concretely: oQe runs calibration samples through the model and records average squared input activation, E[x^2], per quantized Linear input channel.
For MoE SwitchLinear layers, it tracks one importance vector per expert, using router-selected expert indices, so rarely-activated experts still get proper coverage.
Calibration sample count adapts upward from the requested amount when MoE expert coverage isn't yet sufficient.
During affine quantization, scale/bias candidates are chosen to minimize sum(importance * (weight - dequantized_weight)^2) rather than plain unweighted MSE.
If a tensor has no matching imatrix entry, it falls back to standard oQ for that tensor (unless strict coverage is enabled, in which case conversion fails instead of silently falling back).

Table with columns: oQ, oQe (this repo)
	oQ	oQe (this repo)
Bit allocation	Layer-sensitivity mixed precision	Same oQ plan
Affine quantization	Standard min/max affine per group	imatrix-weighted clipping/search per group
Best for	Fast, deterministic conversion with strong baseline quality	Better low-bit retention, especially for MoE and activation-skewed layers

No fine-tuning, distillation, or weight editing was performed beyond this quantization pass; the underlying architecture, tokenizer, and chat template are identical to the base model.

Usage

This is a plain mlx-lm-format checkpoint, so any MLX-based tool works out of the box, no custom loader needed:

bash
pip install mlx-lm

python -m mlx_lm.generate \
  --model programmer-666/Ornith-1.0-35B-oQ8e \
  --prompt "Write a Python function is_prime(n). Keep it short." \
  --max-tokens 512

python
from mlx_lm import load, generate

model, tokenizer = load("programmer-666/Ornith-1.0-35B-oQ8e")
response = generate(model, tokenizer, prompt="Hello", max_tokens=256, verbose=True)
print(response)

For multimodal (image + text) use, the same checkpoint can be loaded with mlx-vlm since the vision encoder is preserved unquantized:

python
from mlx_vlm import load, generate

model, processor = load("programmer-666/Ornith-1.0-35B-oQ8e")
output = generate(model, processor, prompt="Hello", max_tokens=256, temp=1.0, top_p=1.0)
print(output)

It also works directly in LM Studio or oMLX, just search for or point either app at programmer-666/Ornith-1.0-35B-oQ8e.

Ornith is a reasoning model: the response will contain a <think>...</think> block before the final answer. Split on </think> if you only want the final answer.

Benchmarks

Single-request performance vs. context length

BASE (BF16)

Table with columns: Test, TTFT (ms), TPOT (ms), PP (tok/s), TG (tok/s), E2E (s), Throughput (tok/s), Peak Mem (GB)
Test	TTFT (ms)	TPOT (ms)	PP (tok/s)	TG (tok/s)	E2E (s)	Throughput (tok/s)	Peak Mem (GB)
pp1024/tg128	637.0	16.89	1607.4	59.7	2.791	412.8	65.62
pp4096/tg128	2187.3	17.24	1872.6

oQ8e (8-bit)

Table with columns: Test, TTFT (ms), TPOT (ms), PP (tok/s), TG (tok/s), E2E (s), Throughput (tok/s), Peak Mem (GB)
Test	TTFT (ms)	TPOT (ms)	PP (tok/s)	TG (tok/s)	E2E (s)	Throughput (tok/s)	Peak Mem (GB)
pp1024/tg128	603.0	12.34	1698.2	81.7	2.177	529.1	35.38
pp4096/tg128	2262.6	12.60	1810.3

Takeaways:

Memory: oQ8e cuts peak memory by roughly 43 to 46% across all context lengths (e.g. 65.62 GB to 35.38 GB at 1K context; 70.63 GB to 40.18 GB at 200K context).
Generation (TG) speed: oQ8e is about 31 to 37% faster at short-to-mid context (1K to 32K tokens), since decoding is memory-bandwidth bound and benefits directly from the smaller weights. The gap narrows around 64K context and becomes less consistent beyond that.
Prompt processing (PP) speed: Mixed results, oQ8e is slightly faster at very short prompts but 6 to 26% slower at mid-to-large prompts (8K to 131K tokens), which is the expected trade-off of 8-bit compute (dequantization overhead during compute-bound prefill).
Very long context (64K and above): Both TPOT and TG/PP throughput become noticeably less smooth for both models, expected behavior as unified memory pressure and cache effects start to dominate. This range is worth re-validating on your own hardware/config before relying on it.

Batching performance (fixed request size, varying concurrent batch)

BASE (BF16)

Table with columns: Batch, TG (tok/s), Speedup, PP (tok/s), PP per-request (tok/s), TTFT (ms), E2E (s)
Batch	TG (tok/s)	Speedup	PP (tok/s)	PP per-request (tok/s)	TTFT (ms)	E2E (s)
1	59.7	1.00	1607.4	1607.4	637.0	2.791
2	38.1	0.64	506.9	253.4	4040.3

oQ8e (8-bit)

Table with columns: Batch, TG (tok/s), Speedup, PP (tok/s), PP per-request (tok/s), TTFT (ms), E2E (s)
Batch	TG (tok/s)	Speedup	PP (tok/s)	PP per-request (tok/s)	TTFT (ms)	E2E (s)
1	81.7	1.00	1698.2	1698.2	603.0	2.177
2	128.6	1.57	269.5	253.9	3799.6

Takeaways:

oQ8e scales much better with concurrency: at batch size 8 it reaches a 2.45x speedup over its own batch-1 throughput, versus only 1.13x for the BF16 base model. The extra memory headroom freed by 8-bit weights lets more requests be batched before hitting memory/bandwidth limits.
End-to-end latency at batch 8 is roughly 1.8x lower with oQ8e (13.677 s vs 24.681 s), despite doing the same amount of work.
The BASE model actually regresses at batch size 2 (0.64x, worse than single-request), while oQ8e scales up from batch 1 immediately. This suggests the base BF16 model is memory-constrained enough that low-batch concurrency isn't worthwhile on this hardware, whereas oQ8e has enough headroom to benefit from batching right away.

Limitations

Quantization still introduces some numerical approximation vs. the original BF16 weights, even with oQe's sensitivity- and imatrix-aware protections. No formal accuracy/quality regression testing (e.g. on Terminal-Bench 2.1 or SWE-bench) has been run for this specific build; treat throughput/memory numbers above as validated, but re-check task quality for your own use case before production use.
Any tensor without a matching imatrix calibration entry silently falls back to standard oQ precision for that tensor, which is expected but worth knowing if you inspect per-tensor bit allocation.
Prompt-processing (prefill) throughput can be lower than the base model at large context sizes; this build is best suited to workloads dominated by generation rather than very long, one-off prompts.
Behavior beyond approximately 64K context tokens is less predictable for both the base and quantized model and should be validated on your target hardware.

License

Released under the MIT License, inherited from the base model deepreinforce-ai/Ornith-1.0-35B.

Citation

If you use this model, please cite the original Ornith-1.0-35B work:

bibtex
@misc{ornith-35b,
    title = {{Ornith-1.0-35B}: Agentic Coding, Open to All},
    url = {https://deep-reinforce.com/ornith_1_0.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}

And, if useful, this quantized build:

bibtex
@misc{ornith-35b-oq8e,
    title = {{Ornith-1.0-35B-oQ8e}: oMLX Universal Dynamic Quantization (oQe, 8-bit) for Apple Silicon},
    url = {https://huggingface.co/programmer-666/Ornith-1.0-35B-oQ8e},
    author = {{programmer-666}},
    year = {2026}
}

Ornith-1.0-35B-oQ8e

README

Why this exists

About the base model: Ornith-1.0-35B

Quantization method: oMLX Universal Dynamic Quantization

Usage

Benchmarks

Single-request performance vs. context length

Batching performance (fixed request size, varying concurrent batch)

Limitations

License

Citation

Explore FriendliAI today

README

Why this exists

About the base model: Ornith-1.0-35B

Quantization method: oMLX Universal Dynamic Quantization

Usage

Benchmarks

Single-request performance vs. context length

Batching performance (fixed request size, varying concurrent batch)

Limitations

License

Citation