GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
GotoAI-Inc
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
GotoAI-Inc
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
Weights were quantized from bfloat16 to int8, group size 128, symmetric, weight-only
(activations stay 16-bit) using llmcompressor.model_free_ptq. No calibration data was
used and the model was never loaded — the quantizer operates directly on the safetensors.
Architecture, tokenizer, chat template and processor configs are the vendor's, unmodified.
496 Linear modules were converted, covering 78.3% of the checkpoint's bytes:
| component | precision | size |
|---|---|---|
| language-model linears (64 layers) | int8 g128 | 24.73 GB (78.3%) |
embed_tokens + lm_head (untied) | bfloat16 | 5.09 GB (16.1%) |
vision tower (model.visual) | bfloat16 | 0.92 GB (2.9%) |
MTP speculator head (mtp.*) | bfloat16 | 0.85 GB (2.7%) |
| conv1d kernels, norms, biases | bfloat16 | 0.004 GB |
| total | 31.59 GB |
Four things are deliberately left at 16-bit:
model.visual.* — vLLM builds multimodal towers with quant_config=None, so a
checkpoint carrying quantized vision weights cannot be loaded.mtp.* — the built-in multi-token-prediction speculator head, loaded through vLLM's
speculative-decoding path rather than the main stack.linear_attn.conv1d — 3-D causal-convolution kernels in the gated-delta-net blocks,
shape (10240, 1, 4). Not Linear layers, and quantizers reject them outright.lm_head + embed_tokens — precision-sensitive, and lm_head is untied here.The linear-attention projections (in_proj_*, out_proj) are quantized; only the
convolution kernels beside them are excluded.
Runs on released vLLM — the architecture has been supported since 0.25.1, so no nightly build is required:
bash
vllm serve GotoAI-Inc/Qwen3.8-27B-W8A16 \--max-model-len 65536 \--enable-auto-tool-choice --tool-call-parser qwen3_xml \--reasoning-parser qwen3
Do not pass --quantization; compressed-tensors is detected from config.json. The int8
W8A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.
31.59 GB of weights leave the rest of the card for the KV cache. With 4 KV heads at
head_dim 256 and only 16 of the 64 layers using full attention, the KV cache costs
~64 KB/token — about 2 GB at 32k and 4 GB at 64k. On a 48 GB card that is comfortable
at 64k and beyond; on a 40 GB card, budget for roughly 32k, or add --language-model-only
(frees the ~0.92 GB vision tower plus its profiling headroom) for more. This is arithmetic
from config.json, not a measured deployment.
The chat template defaults to reasoning_effort='xhigh', which produces long deliberation.
Both knobs below are template variables, passed through chat_template_kwargs:
json
{"chat_template_kwargs": {"reasoning_effort": "low"}} // xhigh (default) | medium | low{"chat_template_kwargs": {"enable_thinking": false}} // skip thinking entirely
Set a server-wide default with
--default-chat-template-kwargs '{"reasoning_effort": "low"}'; request-level values still
win. preserve_thinking: false drops earlier turns' thinking from history, which matters
for long multi-turn sessions.
262144 tokens natively. The base model card documents a YaRN recipe for 1M tokens via
--hf-overrides plus VLLM_ALLOW_LONG_MAX_MODEL_LEN=1; that is not configured here, and
RoPE scaling costs quality at short contexts, so enable it only if you need it.
Built with llm-quantizer:
bash
./llmq.py run --profile qwen3.8-27b
W8A16 is the profile's default scheme, so no --scheme flag is needed. The command is
equivalent to:
python
# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,# transformers==5.15.1, torch==2.13.0from llmcompressor import model_free_ptqmodel_free_ptq(model_stub="Qwen/Qwen3.8-27B",save_directory="Qwen3.8-27B-W8A16",scheme="W8A16",ignore=["re:.*visual.*", "re:.*mtp.*", "re:.*\\.conv1d$","lm_head", "re:.*embed_tokens.*"],device="cuda:0",)
The source ships as 18 shards of ~4 GB, and a job holds one shard at a time, so the build peaks at a few GB of VRAM — no re-sharding needed and no large GPU required.
No benchmarks have been run. Data-free round-to-nearest quantization degrades quality more than a calibrated (GPTQ/AWQ) or QAT build; how much, for your task, is unmeasured here. Int8 degrades far less than int4 — that is the reason this build exists — but "less" is not "none". Treat the published Qwen3.8 numbers as describing the bfloat16 model, not this one.
For an agentic model the informative checks are well-formed reasoning_content and clean
multi-step tool calls rather than perplexity: structured emission degrades before fluency
does.
Apache 2.0, inherited from the base model — the vendor's LICENSE is included unmodified.
"Qwen" is Alibaba's mark; this repository is not endorsed by or affiliated with Alibaba.