GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
schoggie
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
schoggie
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
On a bandwidth-rich but compute-limited card, batch-1 decode is bound by streaming the weights, so activation precision buys nothing there. The moment you batch — or prefill — the dequantising GEMM becomes the bottleneck, and INT8 activations relieve it.
Measured on one CMP 170HX, vs the same model as INT4 AWQ (W4A16), vLLM, FlashInfer, fp8 KV:
| workload | INT4 (W4A16) | W4A8 | delta |
|---|---|---|---|
| Prefill @8K | 1884 tok/s | 2539 tok/s | +35% |
| Prefill @32K | 1657 tok/s | 2171 tok/s | +31% |
| Decode np=1 | 21.53 ms/step | 21.82 ms/step | −1% (wash) |
| Decode np=4 | 34.78 ms/step | 32.16 ms/step | +8% |
| Decode np=8 | 51.59 ms/step | 45.09 ms/step | +14% |
| Decode np=16 | 88.33 ms/step | 70.88 ms/step | +25% aggregate |
The gain scales with batch size and is ~zero at np=1 — exactly what you expect when the fix relieves a compute-bound GEMM rather than a memory-bound one. Long-context end-to-end (20 × 232K-token prompts) came in ~10% faster than INT4.
The full model.visual.* tower is shipped in bf16, exactly as in the upstream INT4
release: the vision encoder is in quantization_config.ignore, so only the language model
is W4A8. Architecture is Qwen3_5ForConditionalGeneration, and preprocessor_config.json /
video_preprocessor_config.json are included, so image and video inputs work out of the box.
Verified on a single CMP 170HX under vLLM: the ViT is instantiated
(MMEncoderAttention), the LM loads through
markdown
MarlinLinearKernel for CompressedTensorsWNA8Int
Head-to-head against the INT4 AWQ model, same server config, fp8 KV cache:
| gate | INT4 AWQ | W4A8 |
|---|---|---|
| Needle-in-a-haystack @256K, 20 depths | 20/20 | 20/20 |
| HumanEval pass@1 (greedy) | 93.3% | 92.1% |
| Agentic tool-use, 7 build/verify/repair tickets | 144/144 checks | 144/144 |
The HumanEval delta is 2 problems out of 164 — not statistically significant (McNemar on the discordant pairs is nowhere near it), though 10 problems do flip outcome, so the numerics genuinely differ.
Note the ignore list leaves the linear_attn layers unquantized (194 of 195 ignored
modules). Qwen3.8-27B is a hybrid model — only 16 of its 64 layers are full attention, the
rest are linear attention carrying a recurrent state. Quantizing activations there risks
error accumulating along the sequence; a sibling linear-attention model lost ~30 points of
NIAH under W8A8. This checkpoint deliberately does not touch them, and the NIAH result above
suggests that is the right call. Do not "improve" it by quantizing those layers.
vLLM currently hardcodes the activation dtype when building the kernel config, so the
INT8 activation scheme is silently dropped and every mp_linear kernel runs plain
W4A16. There is no error and no warning — you just get INT4 speed and wonder why.
In vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_wNa8.py
(~line 144, inside CompressedTensorsWNA8Int.create_weights):
python
# before — activations silently ignoredact_type=params_dtype,# afteract_type=(torch.int8 if self.input_quant is not Noneand self.input_quant.num_bits == 8 else params_dtype),
With act_type=torch.int8, is_a_8bit becomes true and Marlin's int8 branch arms
(marlin_act_int8_process_scales). Keep Marlin enabled — do not set
VLLM_DISABLED_KERNELS.
Verified present in 0.27.1 and still present in 0.28.1rc1 (both dev87 and dev337).
Easiest application, no rebuild — mount the patched file over the original:
bash
podman run -d --name vllm \-v /path/to/patched/compressed_tensors_wNa8.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_wNa8.py:ro \-v /path/to/models:/models \vllm/vllm-openai:latest \--model /models/qwen38-w4a8 \--max-model-len 256000 --max-num-seqs 16 \--attention-backend FLASHINFER --kv-cache-dtype fp8 \--enable-prefix-caching
Do not trust the log line. Using MarlinLinearKernel for CompressedTensorsWNA8Int
is printed whether or not the int8 branch armed. Check the prefill throughput instead:
Or just read the running file: sed -n 144p on the path above.
num_speculative_tokens=7) reached a
mean acceptance length of 5.00 against this W4A8 target — higher than the 4.14 the same
draft achieved against the INT4 target.int-quantized format variant of this recipe is a dead end: it routes to a kernel that
expects packed weights while int-quantized stores them unpacked. Use pack-quantized.