XReyRobert

Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256

Source And Credits

Source model:

deepreinforce-ai/Ornith-1.0-35B

Quantization tooling and reference recipe:

Thanks to deepreinforce-ai for the Ornith release, to modelcloud for GPTQModel, and to groxaxo for GPTQ-Pro and the Qwen3.6 GPTQ-Pro recipe family this run was aligned with.

Artifact Summary

Table with columns: Field, Value
Field	Value
Source model	`deepreinforce-ai/Ornith-1.0-35B`
Architecture	`Qwen3_5MoeForConditionalGeneration`
Model type	`qwen3_5_moe`
Hidden layers	`40`
Hidden size	`2048`
Experts	`256`

The source config contains mtp_num_hidden_layers=1, but the uploaded weight index contains no mtp.* tensors. Treat this checkpoint as non-MTP for speculative decoding unless a follow-up artifact restores and validates real MTP tensors.

Vision configuration is present from the source architecture, but multimodal serving has not been validated for this quantized release. The intended use case is text generation and coding-agent evaluation.

Quantization Recipe

Table with columns: Setting, Value
Setting	Value
Method	GPTQ-Pro / GPTQModel
Quantizer	`gptqmodel:6.1.0-dev`
Bits	`4`
Group size	`128`
Symmetric quantization	`true`
Desc act	`false`
True sequential

Excluded from quantization by dynamic rules:

embeddings
lm_head
MTP modules
norms
vision / visual modules

The calibration data itself is not included in this model repository.

Intended Serving Shape

This checkpoint is intended for advanced users testing text-only vLLM or GPTQ-compatible serving for Qwen/Ornith MoE checkpoints.

A starting vLLM shape for long-context text serving:

bash
vllm serve XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 \
  --served-model-name ornith-1.0-35b-gptq-pro-foem-4bit-g128-ns256-ctx262k \
  --language-model-only \
  --dtype float16 \
  --quantization gptq_marlin \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --max-num-seqs 1 \
  --kv-cache-dtype fp8_e5m2 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code

Serving context for the published Smoke24/vLLM measurements:

The Smoke24/vLLM numbers were collected on an internal llm-residency vLLM deployment. The custom image recipe is not published yet, so this card does not present that image as a public reproduction target. The stable serving knobs captured from the run are listed for context.

Table with columns: Field, Value
Field	Value
Nomad job profile	`vllm-ornith-35b-gptq-pro-262k`
Served model name	`ornith-1.0-35b-gptq-pro-foem-4bit-g128-ns256-ctx262k`
Critical flags	`--dtype float16`, `--quantization gptq_marlin`, `--kv-cache-dtype fp8_e5m2`, `--reasoning-parser qwen3`, `--tool-call-parser qwen3_xml`, `--max-model-len 262144`, `--max-num-batched-tokens 2096`

Treat this as a serving starting point, not a compatibility guarantee for every vLLM release. GPTQ-Marlin, Qwen3.5 MoE handling, FP8 KV cache, and tool parser behavior are loader-version sensitive.

Public vLLM Reproducibility

This artifact has a public reproducibility path on the unmodified upstream vLLM OpenAI image:

image: docker.io/vllm/vllm-openai:nightly-7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8
vLLM version observed in validation: 0.20.1rc1.dev16+g7a1eb8ac2
GPU class: single RTX 3090 24 GB / Ampere
--enforce-eager was not used
no local sleep/wake patch or localhost/*sleepwake* image is required for the validation below

Validated serving shape:

context: --max-model-len 262144 with --gpu-memory-utilization 0.96
--language-model-only, --dtype float16, --quantization gptq_marlin
--kv-cache-dtype fp8_e5m2, --enable-prefix-caching, --max-num-seqs 1
--max-num-batched-tokens 2096, --max-cudagraph-capture-size 32
--reasoning-parser qwen3, --tool-call-parser qwen3_xml

The 262k profile is tight on 24 GB GPUs; gpu_memory_utilization=0.95 was short on KV cache in validation, while 0.96 passed.

vLLM RTX 3090 / Ampere Workload Snapshot

The image above reports observed vLLM / Prometheus metrics for the served model label ornith-1.0-35b-gptq-pro-foem-4bit-g128-ns256-ctx262k over the recent workload window that included validation traffic:

Table with columns: Metric, Value
Metric	Value
Context budget	`262k`
Decode-only throughput from TPOT	`~151 tok/s`
Prefill throughput	`~4,441 tok/s`
Prefix cache hit ratio	`89.5%`
Average TTFT	`0.63s`
Average E2E latency

These are serving metrics, not a standalone quality benchmark. They describe the observed vLLM GPTQ-Marlin runtime behavior for this deployment shape.

MMLU-Pro 350 Selected Subset

Terminal-Bench 2.0 Smoke24

Terminal-Bench 2.0 Smoke24 is a fixed 24-task coding-agent comparison corpus. It is useful for fast regression and local serving comparison, but it is not a full Terminal-Bench leaderboard submission.

XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 used the long-context card-validation shape: max_model_len=262144, max_input_tokens=220000, 30 minute task timeout, 32 CPU / 48 GiB sandbox, thinking_token_budget=32768, max_output_tokens=40000, temperature 1.0, top-p 0.95, top-k 20, and preserve_thinking=true.

Table with columns: Run, Score, Success rate, Wall-time, Output tokens, Observed decode, LLM API time
Run	Score	Success rate	Wall-time	Output tokens	Observed decode	LLM API time
`ornith-1.0-35b-gptq-pro-foem-4bit-g128-ns256-ctx262k`	`17/24`	`70.8%`	`186.4m`	`498.2k`	`137.6 tok/s`

Smoke24 task list and harness summary:

benchmarks/terminal-bench-2.0/smoke24_task_list_20260616.md

Validation Status

Completed:

Source model loaded for quantization.
Code-oriented calibration mix generated.
GPTQ-Pro FOEM quantization completed.
Final checkpoint saved as five safetensors shards.
Hugging Face repo file list verified after upload.
vLLM serving metrics collected through Prometheus.
Terminal-Bench 2.0 Smoke24 validation run completed.

Not yet validated:

Full Terminal-Bench leaderboard submission.
Vision or multimodal serving.
MTP/speculative decoding.
Broad multi-GPU serving matrix.

Limitations

Experimental quantization.
Smoke24 is a small 24-task slice; read one-task differences with caution.
The calibration mix is code-heavy and was chosen for coding-agent use, not as a general-purpose calibration corpus.
Treat this release as non-MTP for speculative decoding.
Vision support is not validated.
Loader behavior may vary across vLLM, Transformers, GPTQModel, and GPTQ-Marlin versions.

Files

Key files:

model.safetensors.index.json
model-00001-of-00005.safetensors through model-00005-of-00005.safetensors
config.json
generation_config.json
processor_config.json
quantize_config.json
tokenizer.json
tokenizer_config.json

References

Source model: deepreinforce-ai/Ornith-1.0-35B
GPTQModel: modelcloud/gptqmodel
GPTQ-Pro tooling: groxaxo/GPTQ-Pro
Reference recipe: groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit
Terminal-Bench: laude-institute/terminal-bench

Individual Project Notice

This repository is an individual research project. It is not affiliated with, sponsored by, or endorsed by any employer or organization.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

XReyRobert

Model Tree

Base

deepreinforce-ai/Ornith-1.0-35B

Quantized

this model

Input Modalities

Text

Image

Video

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Source And Credits

Source model:

deepreinforce-ai/Ornith-1.0-35B

Quantization tooling and reference recipe:

Thanks to deepreinforce-ai for the Ornith release, to modelcloud for GPTQModel, and to groxaxo for GPTQ-Pro and the Qwen3.6 GPTQ-Pro recipe family this run was aligned with.

Artifact Summary

Table with columns: Field, Value
Field	Value
Source model	`deepreinforce-ai/Ornith-1.0-35B`
Architecture	`Qwen3_5MoeForConditionalGeneration`
Model type	`qwen3_5_moe`
Hidden layers	`40`
Hidden size	`2048`
Experts	`256`

Quantization Recipe

Table with columns: Setting, Value
Setting	Value
Method	GPTQ-Pro / GPTQModel
Quantizer	`gptqmodel:6.1.0-dev`
Bits	`4`
Group size	`128`
Symmetric quantization	`true`
Desc act	`false`
True sequential

Excluded from quantization by dynamic rules:

embeddings
lm_head
MTP modules
norms
vision / visual modules

The calibration data itself is not included in this model repository.

Intended Serving Shape

This checkpoint is intended for advanced users testing text-only vLLM or GPTQ-compatible serving for Qwen/Ornith MoE checkpoints.

A starting vLLM shape for long-context text serving:

bash
vllm serve XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 \
  --served-model-name ornith-1.0-35b-gptq-pro-foem-4bit-g128-ns256-ctx262k \
  --language-model-only \
  --dtype float16 \
  --quantization gptq_marlin \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --max-num-seqs 1 \
  --kv-cache-dtype fp8_e5m2 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code

Serving context for the published Smoke24/vLLM measurements:

Table with columns: Field, Value
Field	Value
Nomad job profile	`vllm-ornith-35b-gptq-pro-262k`
Served model name	`ornith-1.0-35b-gptq-pro-foem-4bit-g128-ns256-ctx262k`
Critical flags	`--dtype float16`, `--quantization gptq_marlin`, `--kv-cache-dtype fp8_e5m2`, `--reasoning-parser qwen3`, `--tool-call-parser qwen3_xml`, `--max-model-len 262144`, `--max-num-batched-tokens 2096`

Treat this as a serving starting point, not a compatibility guarantee for every vLLM release. GPTQ-Marlin, Qwen3.5 MoE handling, FP8 KV cache, and tool parser behavior are loader-version sensitive.

Public vLLM Reproducibility

This artifact has a public reproducibility path on the unmodified upstream vLLM OpenAI image:

image: docker.io/vllm/vllm-openai:nightly-7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8
vLLM version observed in validation: 0.20.1rc1.dev16+g7a1eb8ac2
GPU class: single RTX 3090 24 GB / Ampere
--enforce-eager was not used
no local sleep/wake patch or localhost/*sleepwake* image is required for the validation below

Validated serving shape:

context: --max-model-len 262144 with --gpu-memory-utilization 0.96
--language-model-only, --dtype float16, --quantization gptq_marlin
--kv-cache-dtype fp8_e5m2, --enable-prefix-caching, --max-num-seqs 1
--max-num-batched-tokens 2096, --max-cudagraph-capture-size 32
--reasoning-parser qwen3, --tool-call-parser qwen3_xml

The 262k profile is tight on 24 GB GPUs; gpu_memory_utilization=0.95 was short on KV cache in validation, while 0.96 passed.

vLLM RTX 3090 / Ampere Workload Snapshot

Table with columns: Metric, Value
Metric	Value
Context budget	`262k`
Decode-only throughput from TPOT	`~151 tok/s`
Prefill throughput	`~4,441 tok/s`
Prefix cache hit ratio	`89.5%`
Average TTFT	`0.63s`
Average E2E latency

These are serving metrics, not a standalone quality benchmark. They describe the observed vLLM GPTQ-Marlin runtime behavior for this deployment shape.

MMLU-Pro 350 Selected Subset

Terminal-Bench 2.0 Smoke24

Table with columns: Run, Score, Success rate, Wall-time, Output tokens, Observed decode, LLM API time
Run	Score	Success rate	Wall-time	Output tokens	Observed decode	LLM API time
`ornith-1.0-35b-gptq-pro-foem-4bit-g128-ns256-ctx262k`	`17/24`	`70.8%`	`186.4m`	`498.2k`	`137.6 tok/s`

Smoke24 task list and harness summary:

benchmarks/terminal-bench-2.0/smoke24_task_list_20260616.md

Validation Status

Completed:

Source model loaded for quantization.
Code-oriented calibration mix generated.
GPTQ-Pro FOEM quantization completed.
Final checkpoint saved as five safetensors shards.
Hugging Face repo file list verified after upload.
vLLM serving metrics collected through Prometheus.
Terminal-Bench 2.0 Smoke24 validation run completed.

Not yet validated:

Full Terminal-Bench leaderboard submission.
Vision or multimodal serving.
MTP/speculative decoding.
Broad multi-GPU serving matrix.

Limitations

Experimental quantization.
Smoke24 is a small 24-task slice; read one-task differences with caution.
The calibration mix is code-heavy and was chosen for coding-agent use, not as a general-purpose calibration corpus.
Treat this release as non-MTP for speculative decoding.
Vision support is not validated.
Loader behavior may vary across vLLM, Transformers, GPTQModel, and GPTQ-Marlin versions.

Files

Key files:

model.safetensors.index.json
model-00001-of-00005.safetensors through model-00005-of-00005.safetensors
config.json
generation_config.json
processor_config.json
quantize_config.json
tokenizer.json
tokenizer_config.json

References

Source model: deepreinforce-ai/Ornith-1.0-35B
GPTQModel: modelcloud/gptqmodel
GPTQ-Pro tooling: groxaxo/GPTQ-Pro
Reference recipe: groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit
Terminal-Bench: laude-institute/terminal-bench

Individual Project Notice

This repository is an individual research project. It is not affiliated with, sponsored by, or endorsed by any employer or organization.

Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256

README

Source And Credits

Artifact Summary

Quantization Recipe

Intended Serving Shape

Public vLLM Reproducibility

vLLM RTX 3090 / Ampere Workload Snapshot

MMLU-Pro 350 Selected Subset

Terminal-Bench 2.0 Smoke24

Validation Status

Limitations

Files

References

Individual Project Notice

Explore FriendliAI today

README

Source And Credits

Artifact Summary

Quantization Recipe

Intended Serving Shape

Public vLLM Reproducibility

vLLM RTX 3090 / Ampere Workload Snapshot

MMLU-Pro 350 Selected Subset

Terminal-Bench 2.0 Smoke24

Validation Status

Limitations

Files

References

Individual Project Notice