GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
ghostchoir
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
ghostchoir
Model Tree
Qwen/Qwen3-1.7B
Input Modalities
Output Modalities
Supported Functionality
| Base model | Qwen/Qwen3-1.7B (BF16, 28 blocks, hidden 2048) |
| Scheme | weight-only, 4-bit, group size 128, asymmetric (zero point) |
| Scope | all self_attn / mlp linears in decoder blocks |
| Search | AWQ scale exponent (20-point grid) + clipping (stock per-output-channel search) |
The candidate is scored by the MSE at the output of the block being quantized:
markdown
inps ──► block i (candidate weights) ──► MSE vs block i unquantized
Note this is a block-wise objective. Stock AWQ scores at the output of the
sub-module it perturbs (self_attn, mlp, or a single nn.Linear), which is a
narrower scope than the whole block. Clipping is stock AWQ's per-output-channel
threshold search.
| Dataset | sumin99/Openr1-dataset-sampled-CMA |
| Samples | first 4096 rows (of 32768) |
| Length | first 2048 tokens per row, one document per row (no concatenation) |
| Padding | right-padded with `< |
| Totals | 8,131,131 real tokens, 3.07% padding, 3600/4096 rows truncated |
Text is tokenized as-is — no chat template applied, no stripping (rows already carry ChatML markup).
Perplexity at sequence length 2048. Bold is this model.
| model | WikiText-2 | Δ | OpenR1-CMA held-out | Δ |
|---|---|---|---|---|
| Qwen3-1.7B BF16 (unquantized) | 16.7162 | — | 2.2045 | — |
| RTN W4/g128 (no AWQ) | 20.4892 | +3.7729 | 2.4442 | +0.2397 |
| AWQ W4/g128 block-wise (d=0) | 17.8742 | +1.1580 | 2.3001 |
Held-out slice = rows 4096–5119 of the calibration dataset (never seen by the search). RTN = round-to-nearest group quantization with no AWQ search.
Comparability caveat. The
d=0andd=1arms differ in two ways, not one: the loss scope (block output vs next-block output) and the clipping search (stock per-output-channel vs a single global shrink scored look-ahead). The clipping difference does more work at 3-bit than at 4-bit, so the gap between these two arms should not be read as an ablation of the loss alone.
| AWQ W4/g128 look-ahead (d=1) | 17.1598 | +0.4436 | 2.3239 | +0.1194 |