GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
ulkaa
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
ulkaa
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
The 1m revision contains the same weights as main. Only config.json
changes: YaRN factor 4 extends the original 262,144-token configuration to
1,048,576 tokens.
python
from huggingface_hub import snapshot_downloadsnapshot_download("ulkaa/Qwen3.8-27B-AWQ-INT4",revision="1m",local_dir="models/Qwen3.8-27B-AWQ-INT4-1m",)
The current TP4 production recipe uses:
text
--context-length 1048576--max-total-tokens 1048576--max-mamba-cache-size 40--max-running-requests 64
This produces an exact 1,048,576-token target pool with eight effective speculative request slots. Requests beyond active capacity queue. All active requests share the target pool, and each request's input plus requested output must fit the context limit. A full 1,048,576-token input leaves no room for generated tokens.
Pull the model-scoped image:
bash
docker pull rahulunair/sglang-xpu:qwen3.8-27b-20260819
The copy-paste TP4 commands, required container permissions, readiness checks,
and full context notes live in the
Qwen3.8-27B Docker Hub section.
The moving model alias is rahulunair/sglang-xpu:qwen3.8-27b; use the dated
tag above when reproducibility matters.
The image carries the qualified Intel XPU defaults for decode INT8, symmetric
all-reduce, speculative verify, and graph capture. Do not add SGLang's CUDA
custom collective; the published command uses --disable-custom-all-reduce
so the XPU plugin owns the collective path.
Both supported modes use target verification. The draft proposes tokens; it does not bypass the target model.
The MTP head is inside this checkpoint and needs no second model. It is kept in BF16 and excluded from compressed-tensors quantization.
bash
--speculative-algorithm EAGLE \--speculative-num-steps 7 \--speculative-eagle-topk 1 \--speculative-num-draft-tokens 8
Download the external draft and mount it at /draft:
python
from huggingface_hub import snapshot_downloadsnapshot_download("incoai/Qwen3.8-27B-DFlash2",local_dir="models/Qwen3.8-27B-DFlash2",)
bash
--speculative-algorithm DFLASH \--speculative-draft-model-path /draft \--speculative-num-draft-tokens 8 \--speculative-dflash-block-size 8 \--speculative-draft-window-size 2048
The published image includes the pinned DFlash2 model class and a compact physical draft-KV ring, so the target can retain a 1,048,576-token pool instead of allocating a second full-context draft pool. Radix prefix caching remains enabled. DFlash2 logprob-returning requests are rejected before scheduling because that request shape is not implemented.
Qwen3.8-27B is a hybrid: 48 of its 64 decoder layers use Gated DeltaNet
(linear_attn.*) and 16 use full attention.
The three large DeltaNet projections, in_proj_qkv, in_proj_z, and
out_proj, are quantized. Together they occupy 10.36 GiB in BF16 and 2.73 GiB
in this artifact. The vision tower, MTP head, lm_head, embed_tokens, all
norms, and the small DeltaNet scalar gates in_proj_a and in_proj_b remain
BF16.
| storage view | result |
|---|---|
| 24.33B parameters stored at four bits | 4.16 bits each including group scales and zero points |
| 3.45B parameters kept at BF16 | 12.4% of the model |
| whole artifact | 5.63 bits per parameter |
Activations remain BF16, hence W4A16. The artifact is deliberately larger than a maximally compressed four-bit build because the output head, embeddings, vision tower, MTP head, norms, and small gates stay at full precision.
Qwen/Qwen3.8-27B BF16, not the FP8 release.codeparrot/self-instruct-starcoder and 40% general instructions from
HuggingFaceH4/ultrachat_200k.Qwen3_5DecoderLayer, so each layer was calibrated
against the quantized output of the preceding layers.The group-128 choice was measured against group 32. Smaller groups add scale and zero-point metadata to every weight read. Prefill perplexity moved from 8.87 in BF16 to 9.33 after quantization.
Greedy text can diverge from BF16, as expected for a four-bit build. Evaluate the model on your own workload when exact BF16 behavior matters.
Apache 2.0, inherited from the base model.