GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
quocbao747
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
quocbao747
Model Tree
ornith-ai/Ornith-1.5-9B
Input Modalities
Output Modalities
Supported Functionality
| Method | AWQ via llm-compressor 0.13.0 |
| Format | compressed-tensors, pack-quantized |
| Scheme | 4-bit weights / fp16 activations, asymmetric, group size 128 |
| Quantized | 248 language-model linear layers, including the gated-delta-net projections |
| Full precision | vision tower, output head, token embeddings |
| Calibration | 108,781 tokens distilled from qwen3.8-flash-next, in 53 packed sequences of 2048, chat-templated with reasoning blocks |
The output head and embeddings stay in fp16 because the vocabulary is ~248k tokens, making
each of them roughly a billion parameters on its own — that is why the result is 8.1 GB
rather than the ~5 GB a naive estimate suggests. recipe.yaml records the exact recipe.
Calibration data provenance. Rather than a generic web corpus, the calibration set was
distilled from qwen3.8-flash-next, a Qwen3.8-Flash-Next-series reasoning model served
over an OpenAI-compatible endpoint. Prompts were composed locally across forty topic areas
(debugging, SQL, systems, algorithms, translation, creative writing, tool selection and
others) crossed with eight styles from one-liners to long pasted code and logs; that model
answered each one, and the resulting conversations were rendered through this model's own chat
template with the reasoning block populated, so the calibration text resembles what Ornith
actually emits at inference — thinking sections included. Samples were concatenated and cut
into fixed 2048-token sequences rather than padded, so the activation statistics AWQ uses to
pick its scales are not dominated by padding positions.
vLLM's compressed-tensors 4-bit path requires capability 75 ("Turing and up"), so a T4 is the oldest supported card. Ampere or newer is the comfortable target.
Volta does not work. Tested on a Tesla V100 16GB (capability 7.0): current vLLM and PyTorch builds ship no Volta GPU code at all, so the engine fails while allocating its first GPU buffer, before the quantization check is even reached. No vLLM release works — the last one with Volta support predates support for this architecture by six releases.
Serve with vLLM. On Turing, four settings are required:
The first start autotunes Triton kernels for several minutes and caches the result, so give any process supervisor a generous startup grace period.
Weights use 8.1 GB, leaving a 3.93 GiB KV cache (115,400 tokens). Streamed, fixed output length, unique prompt prefixes so prefix caching cannot flatter prefill.
Decode tokens/s per stream:
| prompt | conc 1 | conc 2 | conc 4 | conc 8 | conc 16 |
|---|---|---|---|---|---|
| 128 | 34.2 | 33.6 | 31.6 | 28.5 | 28.4 |
| 512 | 33.5 | 31.0 | 28.5 | 22.4 | 21.8 |
| 2048 | 32.6 | 27.2 |
Median time to first token, ms:
| prompt | conc 1 | conc 2 | conc 4 | conc 8 | conc 16 |
|---|---|---|---|---|---|
| 128 | 167 | 262 | 564 | 1067 | 3347 |
| 512 | 555 | 836 | 2160 | 2858 | 6932 |
| 2048 | 2537 | 4653 |
Measured on this quantized build and compared against the figures published on the base model card. Sampling followed the base card's general-task recommendation exactly (temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 1.5), with a 16,384-token completion budget.
| Benchmark | This build (AWQ W4A16) | Original (bf16) | Delta |
|---|---|---|---|
| GPQA Diamond | 79.8% (mean of 5 runs, 790/990 pooled, 95% CI +/-2.5) | 86.4 | -6.6 |
| HLE, text-only subset | 18.7% (28/150, 95% CI +/-6.2) | 14.7 | +4.0 |
On GPQA Diamond the 4-bit build trails the original by 6.6 points, and the 2.5-point interval does not reach zero, so this is a clear and real cost of quantization on graduate-level science QA. Most of that gap is a failure to converge rather than wrong reasoning: 69 of 990 answers (7.0%) were still mid-chain-of-thought at a 28,000-token budget and are scored as incorrect. On the answers that did converge, accuracy was 82.7% (n=921). Under the base card's recommended sampling (temperature 1.0, presence_penalty 1.5) the quantized model occasionally enters runaway reasoning on hard items; whether the bf16 original does so at the same rate is not something this run can tell. The runaway cases cluster heavily in Organic Chemistry: 53 of 69 truncations, which is also the weakest subdomain by accuracy. Individual runs ranged 77.3-82.3%, which is why the base card averages five: a single run can land a couple of points either way. Strongest slices: High-energy particle physics (99%), Quantum Mechanics (96%), Astrophysics (94%); weakest: Organic Chemistry (67%), Genetics (45%).
On HLE the measured value lands slightly above the published figure, which is best read as
"no detectable degradation" rather than an improvement, since the intervals overlap heavily
and the judging differs (below). The judge-free multipleChoice slice, the single most
trustworthy number here, scored 39.4% (13/33). By category, HLE accuracy was highest on
Biology/Medicine (40%) and lowest on Math (10.2%), which is also the largest slice.
HLE grading used Claude Opus 4.6, the same judge model the base model card specifies, so this figure is directly comparable to the published 14.7 rather than merely indicative.
The judging was cross-validated. The identical predictions were also graded by GLM-5.3, a
model sharing no lineage with the judge, the model under test, or its base. The two judges
agreed on 84 of 84 co-judged answers (Cohen's kappa = 1.00) and returned the same overall score. This figure is therefore judge-independent, which matters because HLE grading
is otherwise the softest part of the protocol. Per-item verdicts from both judges are published
in benchmarks/ so the grading can be audited or replaced.
These are not like-for-like reproductions, and the differences all sit on the side of making this build look worse, not better:
Raw per-question records are published in benchmarks/.
Terminal-Bench 2.1 was attempted with the official harbor/terminus-2 harness on the 4 easy tasks plus the shortest medium one. 0 of 5 solved; all five hit the agent timeout.
The cause is structural rather than incidental. An agent loop re-sends its whole transcript each step, so context grows monotonically, and decode throughput on this deployment falls with context: 33 tokens/s at 229 prompt tokens, 26 at 3.6K, 10 at 11K, 5.9 at 22K. Blended over the run that is 19.5 tokens/s, so a task needing ~20,000 agent tokens takes ~1,300 s against a 900 s limit. The timeouts were arithmetically certain in advance.
Faster hardware alone would probably not fix it: the abandoned terminal state showed genuine generation failures at long context, including a corrupted line written into a Coq proof, a syntax error from a truncated identifier, and files created from mangled shell keystrokes. Three of five tasks had run 17-19 steps without converging.
Treat this build as suitable for chat-length reasoning and single-shot QA, not for long-horizon agentic work. The SWE-bench family is further out of reach still: its protocol specifies a 256K context, which needs 8.0 GiB of KV cache on top of 8.1 GiB of weights and cannot fit a 16 GB card at any setting.
Quantizing this model family hits a Transformers/llm-compressor interaction bug that makes
the gated-delta-net layer's forward arguments unrecoverable to the AWQ pass; a one-line fix
in Transformers resolves it. Full method, benchmark methodology and the V100 investigation
are in REPORT.md; raw benchmark data is in
benchmarks/t4_sweep_cells.jsonl.
MIT, inherited from the base model. All credit for the model goes to ornith-ai; this repository contributes only the quantization.
| 17.2 |
| 9.5 |
| 6.7 |
| 8192 | 27.9 | 13.3 | 6.1 | 3.1 | – |
| 16384 | 23.7 | 11.1 | 5.7 | – | – |
| 8178 |
| 13133 |
| 23387 |
| 8192 | 15029 | 23485 | 39595 | 69905 | – |
| 16384 | 43188 | 66668 | 110818 | – | – |