GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
emwesoft
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
emwesoft
Model Tree
ornith-ai/Ornith-1.5-397B
Input Modalities
Output Modalities
Supported Functionality
| Base model | ornith-ai/Ornith-1.5-397B (MIT) |
| Format | EXL3 v1.4.2, --hq mixed precision |
| Average bitrate | 3.51 bpw (excluding head) |
| On-disk size | ~166 GiB (26 safetensors shards) |
| Max context | 262,144 tokens |
| Architecture | Qwen3_5MoeForConditionalGeneration |
| Component | Bits |
|---|---|
| Experts (up/gate/down_proj) | 3 bpw |
| Linear attention (in_proj_qkv / in_proj_z / out_proj) | 5 bpw |
| Full self-attention (q/k/v/o_proj) | 5 bpw |
| Shared expert (all three projections) | 5 bpw |
lm_head | 8 bpw |
| Vision tower | unquantized (BF16) |
| Embeddings / norms | unquantized |
Calibration: 250 rows × 2048 cols · Codebook: mul1 · Output scales: always
60 layers · hidden size 4096 · 32 attention heads / 2 KV heads (GQA) · 512 experts, 10 active per token · MoE intermediate 1024 · vocab 248,320 · mixed linear/full attention (every 4th layer is full attention) · RoPE theta 10M.
Generated directly from the full BF16 weights. This checkpoint was quantized from the
unquantized ornith-ai/Ornith-1.5-397B BF16 source (740 GB, 122 files) — not from a
previously-quantized or GGUF intermediate. The EXL3 --hq encoder calibrates against the
BF16 tensors and assigns each layer the bitrate that minimizes its quantization error, so
this is a single-generation lossy pass from the original precision rather than a re-quant.
The mixed-precision layout is the HQ encoder trading bits where they reduce error the most, against the 192 GB target:
lm_head at 8 bpw. The head maps directly to the output logits over the full 248K
vocab, so any error there distorts the entire token distribution. Keeping it near-full
precision is the cheapest way to protect output quality.Why this fits 192 GB so cleanly: pushing the cheap expert bulk down to 3 bpw and keeping only the error-sensitive dense paths (attention/router/head) at higher precision is what lets the full model, plus a 1,048,576-token FP8 KV pool for four concurrent 262K contexts, land on two 96 GB cards with ~1 GB + ~2 GB to spare. The bits follow the error.
No perplexity, KL divergence, or benchmark-vs-BF16 error was computed for this release. Validation covered structural integrity only, and it passed cleanly: no NaN/Inf tensors, all 60/60 layers present, all 512/512 experts present, tokenizer byte-for-byte identical to the source, and key config values preserved. See the "How well does it work?" section above for the (deliberately modest) quality framing.
Measured and served with ExLlamaV3 (tensor parallel across both cards). See ornith15-exl3-modelcard-perf.md for the full throughput table.
| GPUs | 2 × NVIDIA RTX PRO 6000 Blackwell Workstation (96 GB each) |
| Interconnect | PCIe gen4 x16, no NVLink — TP all-reduce crosses the host bridge |
| KV cache | FP8 (E4M3), cache_mode 8,8 |
| KV pool | 1,048,576 tokens — 4 concurrent full-context (262K) sessions |
| Decode throughput | ~52 tok/s (batch size 1) |
| Time to first token | 0.16 – 0.23 s |
| Driver / CUDA / PyTorch | 610.57.04 / 13.0 / 2.13.0+cu130 |
VRAM footprint: the full checkpoint plus a 1,048,576-token FP8 KV pool (four 262K contexts) fits on this GPU pair with headroom to spare — roughly ~95 GB + ~94 GB in use, leaving about 1 GB free on one card and 2 GB free on the other.
markdown
--host 0.0.0.0 --port 5000 \--model_dir /models/Ornith-1.5-397B-EXL3-HQ-35bpw \--tensor_parallel_size 2 --cache_mode 8,8 \--context_length 262144 --max_cache_size 1048576 \--reasoning_parser qwen3 --tool_call_parser qwen3_xml
Short answer: very well, subjectively — but this is not a benchmarked release.
This quant did well in a short, informal benchmark test I ran, but I have not run — and do not plan to run — a full benchmark suite (no SWE-Bench / Terminal-Bench / perplexity comparison against the unquantized weights is included here). The ~52 tok/s number above is a decode-throughput measurement only, not a quality score. Take the model on its own merits and evaluate it on your own tasks.
<think> block before the final answer. Serve with a --reasoning_parser qwen3 so the chain-of-thought is returned in a separate reasoning_content field.<tool_call> blocks that parse into the standard OpenAI-style tool_calls field — use --tool_call_parser qwen3_xml.Quantization of ornith-ai/Ornith-1.5-397B, released by DeepReinforce AI under the MIT license. This EXL3 quant carries the same license forward. The mixed-precision HQ recipe matches the earlier Ornith-1.0 quant at gigascake/Ornith-397B-EXL3-HQ-35bpw.