GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
ghostchoir
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
ghostchoir
Model Tree
Qwen/Qwen3-1.7B
Input Modalities
Output Modalities
Supported Functionality
| Base model | Qwen/Qwen3-1.7B (BF16, 28 blocks, hidden 2048) |
| Scheme | weight-only, 4-bit, group size 128, asymmetric (zero point) |
| Scope | all self_attn / mlp linears in decoder blocks |
| Search | AWQ scale exponent (20-point grid) + clipping (10-point global shrink grid, look-ahead scored) |
The candidate for block i is scored one block later, at the output of block
i+1, which stays in original precision:
markdown
inps ──► block i (candidate weights) ──► block i+1 (BF16, untouched) ──► MSE
The reference is the same path with block i also unquantized. Block 27 has no
successor, so its loss falls back to its own output.
Because AWQ's scale rewrite (prev_op /= s, W *= s) is algebraically equal to
leaving prev_op alone and using Q(W·diag(s))·diag(1/s), a candidate is
simulated by swapping only the linear's weight, so the measured loss is exactly
the deployed one.
Clipping granularity. The look-ahead loss is a single scalar after block
i+1 and does not decompose over output channels, so clipping searches one
shrink ratio per linear rather than stock AWQ's per-output-channel threshold.
| Dataset | sumin99/Openr1-dataset-sampled-CMA |
| Samples | first 4096 rows (of 32768) |
| Length | first 2048 tokens per row, one document per row (no concatenation) |
| Padding | right-padded with `< |
| Totals | 8,131,131 real tokens, 3.07% padding, 3600/4096 rows truncated |
Text is tokenized as-is — no chat template applied, no stripping (rows already carry ChatML markup).
Perplexity at sequence length 2048. Bold is this model.
| model | WikiText-2 | Δ | OpenR1-CMA held-out | Δ |
|---|---|---|---|---|
| Qwen3-1.7B BF16 (unquantized) | 16.7162 | — | 2.2045 | — |
| RTN W4/g128 (no AWQ) | 20.4892 | +3.7729 | 2.4442 | +0.2397 |
| AWQ W4/g128 block-wise (d=0) | 17.8742 | +1.1580 | 2.3001 | +0.0956 |
Held-out slice = rows 4096–5119 of the calibration dataset (never seen by the search). RTN = round-to-nearest group quantization with no AWQ search.
Comparability caveat. The
d=0andd=1arms differ in two ways, not one: the loss scope (block output vs next-block output) and the clipping search (stock per-output-channel vs a single global shrink scored look-ahead). The clipping difference does more work at 3-bit than at 4-bit, so the gap between these two arms should not be read as an ablation of the loss alone.
| AWQ W4/g128 look-ahead (d=1) | 17.1598 | +0.4436 | 2.3239 | +0.1194 |