GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
whpthomas
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
whpthomas
Model Tree
Base
deepreinforce-ai/Ornith-1.0-35B
Quantized
this model
Input Modalities
Text
Image
Video
Output Modalities
Text
Supported Functionality
Dedicated Endpoints
| Property | Value |
|---|---|
| Base Model | deepreinforce-ai/Ornith-1.0-35B |
| Architecture | Qwen3.5-MoE (Mixture of Experts) |
| Quantization | int4 AutoRound (W4A16) |
| Bits | 4-bit |
| Group Size | 128 |
| Symmetric | Yes |
| Calibration Dataset | NVIDIA OpenCodeInstruct |
| Quantization Tool | spark-auto-round v0.14.3 |
| Context Length | 196,608 tokens |
| Model Size | ~19 GB (quantized) |
This model was quantized using AutoRound, an advanced quantization technique from Intel that uses signed gradient descent to jointly optimize weight rounding and clipping ranges.
markdown
--batch_size 8--iters 1000--nsamples 512--seqlen 2048--dataset opencode-instruct--group_size 128
| Metric | Value |
|---|---|
| Peak RAM | 111.31 GB |
| Peak VRAM | 28.00 GB |
| Layers Passed | 22/40 (55%) |
| Layers Warning | 18/40 (45%) |
The model retains strong quality with all layers meeting minimum thresholds. Layers 22-39 show slightly elevated sensitivity (cosine similarity 0.986-0.990) which is typical for deeper MoE layers.
Note: Shared expert gates are preserved in FP16 to maintain MoE routing accuracy.
python
from transformers import AutoModelForCausalLM, AutoTokenizermodel_path = "whpthomas/Ornith-1.0-35B-int4-AutoRound"# Load tokenizertokenizer = AutoTokenizer.from_pretrained(model_path)# Load quantized modelmodel = AutoModelForCausalLM.from_pretrained(model_path,device_map="auto",torch_dtype="auto",)# Generate textmessages = [{"role": "user", "content": "Explain quantum computing in simple terms."}]text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(text, return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=512)print(tokenizer.decode(outputs[0], skip_special_tokens=True))
For optimal performance, use vLLM with the recommended serving configuration:
bash
pip install vllm
bash
vllm serve whpthomas/Ornith-1.0-35B-int4-AutoRound \--served-model-name ornith-1.0-35b \--max-model-len 196608 \--gpu-memory-utilization 0.55 \--load-format auto \--attention-backend flashinfer \--moe-backend marlin \--enable-prefix-caching \--enable-chunked-prefill
For DGX Spark (GB10) with optimal performance:
bash
vllm serve whpthomas/Ornith-1.0-35B-int4-AutoRound \--served-model-name ornith-1.0-35b \--max-model-len 196608 \--gpu-memory-utilization 0.55 \--max-num-batched-tokens 16384 \--max-num-seqs 8 \--optimization-level 3 \--performance-mode throughput \--load-format instanttensor \--attention-backend flashinfer \--moe-backend marlin \--enable-prefix-caching \--enable-chunked-prefill \--default-chat-template-kwargs '{"preserve_thinking":true}' \--enable-auto-tool-choice \--tool-call-parser qwen3_coder \--reasoning-parser qwen3 \--generation-config auto \--override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":-1,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'
bash
export TORCH_MATMUL_PRECISION=highexport NVIDIA_FORWARD_COMPAT=1export NVIDIA_DISABLE_REQUIRE=1export CUDA_DEVICE_MAX_CONNECTIONS=1export VLLM_MARLIN_USE_ATOMIC_ADD=1export FLASHINFER_DISABLE_VERSION_CHECK=1
This model supports tool calling with the Qwen3 coder parser. When using vLLM with --enable-auto-tool-choice --tool-call-parser qwen3_coder, the model can invoke tools and return structured function calls.
The model supports extended thinking with the Qwen3 reasoning parser. Use --reasoning-parser qwen3 and set preserve_thinking: true in chat template kwargs to enable reasoning traces.
Based on testing with the Qwen3.5-MoE architecture on DGX Spark:
| Metric | Value |
|---|---|
| Throughput (MTP) | ~26-30 t/s |
| Throughput (DFlash) | ~35-40 t/s |
| Latency (Time to First Token) | ~100-200ms |
Performance varies based on context length, batch size, and hardware configuration.
If you use this quantized model, please cite both the base model and the quantization tool:
bibtex
@misc{ornith2025,title={Ornith-1.0-35B},author={DeepReinforce AI},year={2025},publisher={HuggingFace},url={https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B}}@misc{sparkautaround2025,title={Spark Auto Round},author={whpthomas},year={2025},publisher={GitHub},url={https://github.com/whpthomas/spark-auto-round}}
The quantized model inherits the license of the base model. Please refer to the original model card for license details.
This is an int4 AutoRound quantized model. It requires: