GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
barozp
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
barozp
Model Tree
barozp/Qwen3.8-27B-Opus-Distill-v2
Input Modalities
Output Modalities
Supported Functionality
ValueError: Weight output_partition_size = 48 is not divisible by weight quantization block_n = 128,
delete your local copy / re-download so you get the fixed config.json. Root cause: the
modules_to_not_convert list must use bare module paths (...linear_attn.in_proj_ba) --
sglang resolves fused GDN layers through their shard names (in_proj_b, in_proj_a),
so tensor-name-style entries (....weight) never match and a merged projection got FP8'd
despite being bf16 in the checkpoint. Weights were always correct and are unchanged.| What | Where | Result |
|---|---|---|
| Structural integrity | build time | Zero NaN/Inf; vision/MTP tensors byte-identical to the bf16 source; size ratio as expected |
transformers load + generation | A100 (auto-dequantizes to bf16 below compute capability 8.9) | Coherent output; token counts matched the bf16 model on a reasoning-loop repro -- i.e. validated on dequantized-bf16 compute, not native FP8 |
| Native FP8 execution under sglang | RTX PRO 6000 Blackwell Server Edition (sm_120), sglang nightly cu129 stack | Full construction + all 1599 tensors loaded; server came up healthy across five consecutive runs; a real chat completion returned generated text with default sampling parameters |
| MTP / speculative decoding (NEXTN) | same GPU | Draft head (, fp8) loads and serves; the internal warmup request completes through the draft/verify pipeline |
Known environment limitation (not specific to this checkpoint): flashinfer's JIT arch-check currently rejects consumer Blackwell (sm_120) and bites in up to three places: attention planning during CUDA-graph capture, top-k/top-p sampling at request time, and EAGLE verify sampling during speculative decoding. Workarounds are documented below. One consequence on sm_120 specifically: a fully non-greedy MTP-accelerated user request cannot be exercised there today (the EAGLE verify sampler hard-wires flashinfer via sgl_kernel); greedy warmup requests do pass. On Ada/Hopper (sm_89/sm_90) flashinfer works normally and none of these paths are affected. vLLM could not be independently verified in our test image for dependency-conflict reasons unrelated to this checkpoint; the format matches the standard HF FP8 layout used by Qwen's own releases.
Both checkpoints were evaluated under an identical protocol so the delta isolates true quantization cost from any serving-stack numerics:
| Task | Metric | BF16 source | FP8 build | Δ (FP8−BF16) |
|---|---|---|---|---|
| MMLU | acc | 0.8474 | 0.8462 | −0.0012 |
| HellaSwag | acc_norm | 0.7520 | 0.7480 | −0.0040 |
| ARC-Challenge | acc_norm | 0.6240 | 0.6180 |
Every delta sits inside one standard error (~±0.02): the FP8 build's quality cost on these tasks is statistically indistinguishable from zero. Sanity check: our BF16 rerun reproduced this card's published reference values almost exactly (0.8476 / 0.7500 / 0.6220), confirming both the card and the protocol.
lm-evaluation-harness 0.4.12 · transformers 5.15.0 · torch 2.11.0+cu128--model hf) — no serving layer, no batching nondeterminismbfloat16 for both checkpoints · chat template OFF · 0-shot (--num_fewshot 0)--limit 500 (QUICK mode) · --batch_size 16 · --seed 1234Command shape (run once per checkpoint):
bash
python3 -m lm_eval \--model hf \--model_args pretrained=<CHECKPOINT>,dtype=bfloat16,trust_remote_code=True \--tasks mmlu,hellaswag,arc_challenge \--num_fewshot 0 \--batch_size 16 \--limit 500 \--seed 1234 \--output_path <OUT_DIR>
Notes:
pip install "kernels==0.16.0" — otherwise you'll hit markdown
ImportError: finegrained-fp8 kernel unavailable
Standard launch on Ada/Hopper (sm_89 / sm_90 -- e.g. RTX 4090, RTX 6000 Ada, H100):
bash
python3 -m sglang.launch_server \--model-path barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 \--trust-remote-code
Consumer Blackwell cards (sm_120: RTX PRO 6000, RTX 5090) currently need three workaround flags until flashinfer fixes its sm_120 detection:
bash
python3 -m sglang.launch_server \--model-path barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 \--trust-remote-code \--attention-backend triton \--disable-cuda-graph \--sampling-backend pytorch
Optional MTP / speculative decoding (NEXTN) -- add to either command above:
bash
--speculative-algorithm NEXTN \--speculative-num-steps 3 \--speculative-eagle-topk 1 \--speculative-num-draft-tokens 4
(If you are on sm_120, include the three workaround flags here too.)
python
from transformers import AutoModelForCausalLM, AutoTokenizerimport torchmodel = AutoModelForCausalLM.from_pretrained("barozp/Qwen3.8-27B-Opus-Distill-v2-FP8", dtype=torch.bfloat16, device_map="auto",)tokenizer = AutoTokenizer.from_pretrained("barozp/Qwen3.8-27B-Opus-Distill-v2-FP8")
On GPUs below compute capability 8.9, transformers will automatically dequantize to bf16
at load time (you'll see a warning) -- this still works correctly, it just won't give you
the FP8 memory/speed benefit.
Qwen3_5ForCausalLMMTP| Parity check | same GPU | Qwen's own official FP8 release behaves identically through the identical pipeline (same stages, same failure points when the test GPU's environment interferes) |