GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
piscesbody
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
piscesbody
Model Tree
orcarouter/Qwen3.8-27B-Uncensored
Input Modalities
Output Modalities
Supported Functionality
| Component | Format | Notes |
|---|---|---|
| MLP gate/up/down (64 layers) | NVFP4 (W4A4-serialized, group 16) | main size saving |
| lm_head | NVFP4 (group 16) | aggressive; passed our 12-question reasoning suite (see Eval) |
| GDN linear_attn projections (in_proj_qkv/z, out_proj, 48 layers) | FP8 (e4m3, per-tensor) | keeps decode fast on Ada |
| self_attn q/k/v/o projections (16 layers) | FP8 (e4m3, per-tensor) | same |
| embed_tokens, norms, vision tower | BF16 | untouched |
bash
python -m sglang.launch_server \--model-path /path/to/Qwen3.8-27B-Uncensored-NVFP4-v2 \--language-only `# optional: text-only serving skips vision weights` \--tp 1
SGLANG_DISABLE_SILU_FP4_QUANT_FUSION=1 (the fused SiLU+FP4 kernel has no
Ada backend in current flashinfer and will crash startup with
Invalid backend: 89 otherwise).The
modelopt_fp4(W4A4) compressed-tensors builds published elsewhere may refuse to load on Ada (min_capability=100). This checkpoint ships the W4A16-serialized form, which runs on SM89.
MIXED_PRECISION modelopt checkpoints should load like nvidia/Qwen3.8-27B-NVFP4 (vllm ≥ 0.27). Not validated by us — feedback welcome.
| metric | FP8 source | this NVFP4 build |
|---|---|---|
| weights on disk | 29 GB | 21.1 GB (vision included); 20.2 GB text-only |
| KV cache pool @48G | ~310k tokens | ~500k tokens (+61%) |
| single-stream decode (code task) | 73-84 tok/s | 100-125 tok/s |
| cold prefill 14k tok | 4280 tok/s | ~3200 tok/s (Ada has no FP4 tensor cores; Marlin W4A16 tax) |
| reasoning eval | 12/12 | 12/12 |
Decode wins come from FP4 weight bandwidth + FP8 attention projections; prefill pays a Marlin dequant tax on Ada only (Blackwell has native FP4 compute). Measured on one RTX 4090 48G, TP1, DFlash2 speculative decoding K=8, fp8 KV cache.
model-0000{1,2}-of-00003.safetensors — quantized language model (20.2 GB)model-00003-of-00003.safetensors — BF16 vision tower, 333 tensors (0.92 GB, from the base checkpoint)hf_quant_config.json — modelopt MIXED_PRECISION manifest (per-layer recipe)LICENSE — Apache-2.0, inherited from the base checkpointREADME.zh.md — Chinese version of this documentReproduction scripts (see next section):
quantization_reproduce.py — step 1: quantize + exportvision_merge.py — step 2: copy the BF16 vision tower inmake_composite_config.py — step 3: rebuild config.json into the composite VL shapebash
# 1. quantize (mixed precision) -> writes a flat-config exportpython quantization_reproduce.py \--src /path/to/Qwen3.8-27B-Uncensored \--dst /path/to/out# 2. merge the BF16 vision tower (no requant; file-level tensor copy)python vision_merge.py --base /path/to/Qwen3.8-27B-Uncensored --dst /path/to/out# 3. rebuild config.json into the composite VL shapepython make_composite_config.py --base /path/to/Qwen3.8-27B-Uncensored --dst /path/to/out
Step 3 is not optional for SGLang multimodal serving. export_hf_checkpoint
writes a flat text config (architectures: ["Qwen3_5ForCausalLM"]), but the
encoder/decoder split whitelist used by --language-only only accepts the
composite name Qwen3_5ForConditionalGeneration with a nested text_config.
A flat export loads as text-only but is rejected by the split hook.
Two modelopt 0.46.0 gotchas the scripts already handle (both cost a failed run each if you hit them from scratch):
{"algorithm": "max"} is required in mtq.quantize(...). Without it,
calibrate() dispatches to NoneCalibrateMode and silently skips the
forward loop — the run "succeeds" in seconds with every amax unset._amax export bug. The FP8 exporter reads weight_quantizer._amax, but
TensorQuantizer now exposes it as the amax property. quantization_reproduce.py
wraps _export_quantized_weight to rebind it temporarily.Requirements: torch, transformers, modelopt==0.46.0, one CUDA GPU. The BF16
weights are ~54 GB and do not fit a 48 GB card, so device_map spills to host RAM
(--gpu-mem / --cpu-mem control the split; defaults 24GiB / 80GiB).
Apache-2.0, same as the base. Abliteration by orcarouter (see base repo); quantization pipeline by the uploader. If you redistribute, keep the base model attribution.