urakozz

Ornith-1.0-35B-int4-AutoRound

README

License: apache-2.0

Quantization Configuration

Table with columns: Parameter, Value
Parameter	Value
Base Model	`deepreinforce-ai/Ornith-1.0-35B`
Quantization Scheme	`W4A16` (4-bit weights, 16-bit activations)
Quantization Engine	Intel Auto-Round
Group Size	`64`
Calibration Sequence Length	`2048`
Calibration Samples (`nsamples`)	`64`
Optimization Iterations (`iters`)	`300`
Format	`auto_round`

Quickstart & Serving

Ornith-1.0-35B is a reasoning model: by default, the assistant turn opens with a <think> ... </think> block before the final answer. The serving commands below configure the reasoning and tool-call parsers so the chain-of-thought is returned in a separate reasoning_content field.

1. Launch with vLLM

Install vLLM (version >= 0.25.1 recommended):

bash
pip install auto-round vllm

Start the vLLM OpenAI-compatible server:

bash
vllm serve urakozz/Ornith-1.0-35B-int4-Autoround \
    --served-model-name Ornith-1.0-35B-int4-Autoround \
    --quantization auto-round \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 131072 \
    --gpu-memory-utilization 0.90 \
    --enable-auto-tool-choice --tool-call-parser qwen3_xml \
    --reasoning-parser qwen3 \
    --trust-remote-code

2. Querying the OpenAI-Compatible API

Once the vLLM server is running, connect via any OpenAI client:

python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="Ornith-1.0-35B-int4-Autoround",
    messages=[
        {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ],
    temperature=0.6,
    top_p=0.95,
    max_tokens=2048,
)

message = response.choices[0].message

# reasoning_content holds the <think> trace; content holds the final answer
print("Reasoning:", getattr(message, "reasoning_content", None))
print("Answer:", message.content)

Benefits & Memory Footprint

VRAM Savings: Quantizing the 35B model with W4A16 and group_size=64 reduces memory consumption from ~70GB+ down to ~20–22GB VRAM, enabling execution on single consumer/workstation GPUs.
Accuracy: Auto-Round optimizes quantization parameters over 300 iterations per block, significantly mitigating quantization loss compared to standard round-to-nearest (RTN) methods.

Acknowledgments & Citation

Base Model: deepreinforce-ai/Ornith-1.0-35B
Quantization Engine: Intel Auto-Round

bibtex
@misc{ornith-35b,
    title = {{Ornith-1.0-35B}: Agentic Coding, Open to All},
    url = {[https://deep-reinforce.com/ornith_1_0.html](https://deep-reinforce.com/ornith_1_0.html)},
    author = {{DeepReinforce Team}},
    year = {2026}
}

markdown
undefined

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

urakozz

Model Tree

Base

deepreinforce-ai/Ornith-1.0-35B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality