GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
lued
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
lued
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
Qwen3.8-27B is a big model. This build makes it fast on consumer hardware:
Measured on two RTX 3090s: same engine, same 262K context, same prompts, single stream, cold cache. Generation tokens per second.
| Prompt tokens | Autoregressive | MTP4 | DFlash2 | DFlash2 vs MTP4 |
|---|---|---|---|---|
| 128 | 47 | 77 | 117 | 1.5x |
| 2,048 | 47 | 73 | 103 | 1.4x |
| 8,192 | 47 | 75 | 102 | 1.4x |
DFlash2 wins at every prompt length we tested. The reason is the drafter: DFlash2 proposes 7 draft tokens per step and the target verifies them all in one pass. Native MTP proposes 4. More drafts per verification step means more accepted tokens and less idle GPU time.
| BF16 original | This build | |
|---|---|---|
| Size | ~56 GiB | 28 GiB |
| Mean KLD | 0 | 0.0007 |
| Top-1 agreement | 100% | 98.1% |
KLD is a fancy way of asking "does the quantized model pick the same next token as the original?" Lower is closer. A mean of 0.0007 nats means the INT8 weights and the BF16 weights are nearly interchangeable. Halving the size is what lets the full 262K context fit on two 3090s.
Three steps. The full command with every flag is in TECHNICAL.md.
bash
# 1. Models (pulls into ~/.cache/huggingface)hf download lued/Qwen3.8-27B-INT8-W8A16-DFlash2hf download lued/Qwen3.8-27B-DFlash2-W8# 2. Patches (DFlash2 support is not in a released vLLM yet)git clone https://github.com/noonghunna/club-3090.git# 3. Serve (condensed; full command in TECHNICAL.md)export CLUB3090="$HOME/club-3090"podman run --rm --replace --device nvidia.com/gpu=all --ipc=host -p 8080:8080 \-v ~/.cache/huggingface:/root/.cache/huggingface \-v "$CLUB3090/models/qwen3.6-27b/vllm/patches/vllm-pr52816-dflash2":/etc/club3090/pr52816:ro \-v "$CLUB3090/models/qwen3.6-27b/vllm/patches/vllm-pr48375-mamba-drop-eagle-block":/etc/club3090/pr48375:ro \--entrypoint bash docker.io/vllm/vllm-openai:nightly-5a4c8d99242e9e069b604d0e9b969e77f7dd501d \-c 'bash /etc/club3090/pr48375/install.sh || exit 1; bash /etc/club3090/pr52816/install.sh || exit 1; exec vllm serve "$@"' -- \lued/Qwen3.8-27B-INT8-W8A16-DFlash2 \--tensor-parallel-size 2 --max-model-len 262144 --kv-cache-dtype fp8_e4m3 \--speculative-config '{"method":"dflash","model":"lued/Qwen3.8-27B-DFlash2-W8","num_speculative_tokens":7}'
Built, audited, and published 2026-08-19. Both release audits and the drafter audit pass, KLD measured, serving validated at 262K on two 3090s. The drafter ships separately as lued/Qwen3.8-27B-DFlash2-W8. All engineering detail, audit evidence, and measured tables are in TECHNICAL.md.