GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
0xWhiteMage
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
0xWhiteMage
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
Qwen3.8 27B Kearuga is a custom tiered sensitivity hybrid-quantized checkpoint of Qwen/Qwen3.8-27B, specifically engineered for high-throughput and interactive speculative-decoding serving on a single NVIDIA DGX Spark (GB10 / Grace-Blackwell, 128 GB Unified Memory).
By pairing GPTQ with Four-Over-Six (4o6) group scales, NVFP4 AWQ, FP8 E4M3, and protected BF16, Kearuga:
bench/scale.py probe measures C1 44.7 · C2 78–94 · C4 131 tok/s aggregate (T=0, thinking off, 512 forced tokens, aggregate incl. TTFT 0.23–0.34 s; identical with the GPU clock lock on or off), and the paired 50-prompt battery puts it +14.5 % C1 / +10.8 % C4 over the stock z-lab drafter (C1 net-decode medians: code 57.6, math 62.2, tool calls 114.6, prose 21.6 tok/s). Serving stack: GitHub v0.6.6; drafter release v1.0. Without speculative decoding the same target measures 10.30 tok/s C1 / 39.56 tok/s C4 on that battery (serving stack v0.6.7) — a 3.46× / 2.75× gain from the drafter.The table below presents fidelity and quality metrics measured on an isolated NVIDIA DGX Spark (GB10 / SM121, 128 GB Unified Memory) running SGLang (KV cache in BF16, thinking disabled, temperature 0). They do not depend on the drafter — the target verifies every drafted token:
| Metric | Base Model: Qwen/Qwen3.8-27B (Native BF16) | Qwen3.8 27B Kearuga (Hybrid Quantized) | Delta / Operational Significance |
|---|---|---|---|
| Weight Footprint | 51.8 GiB | 24.85 GB | −52.0% (leaves ≈ 90 GB of the 121 GB unified memory for KV cache, drafter and runtime) |
| Fidelity-40 Mean KL Divergence | 0.0000 | 0.0165 | −50.6% KL reduction vs legacy NVFP4 baseline (0.0334) |
| Fidelity-40 Mean Jensen-Shannon (JS) | 0.0000 | 0.0034 |
Kearuga drafter (K=12, NVFP4, 64K draft head). In a paired bake-off on the same image, target, flags and clock — 50-prompt battery (10 prompts × 5 domains) × 2 runs, temperature 0, thinking off; "net decode tok/s" = (completion tokens − 1)/(last − first token time) per request, TTFT excluded — the Kearuga DFlash 2 drafter with the draft-head overlay reached 35.33 tok/s C1 (+14.5 %) and 108.90 tok/s C4 (+10.8 %) aggregate vs the stock drafter (30.85 / 98.31), better on all 10 domain cells. C1 net-decode medians: code 46.27 → 57.62, math 51.21 → 62.18, tool calls 80.82 → 114.63, prose 19.45 → 21.64, instruction-following 19.87 → 21.80. The target still verifies every drafted token, so output is unchanged (Fidelity-40 KL 0.0165, top-1 40/40). Full tables on the drafter model card.
Uniform quantization (applying INT4 or NVFP4 indiscriminately across all layers) degrades attention sink states, vocabulary distribution tails, and speculative draft features.
Inspired by empirical mixed-precision sensitivity research (EXL3 tiered sensitivity), Kearuga partitions the model's 2,009 tensors into four distinct sensitivity tiers:
| Tier | Module Scope | Precision | Count | Architectural Purpose |
|---|---|---|---|---|
| Tier 1 (Protect) | embed_tokens, lm_head, all RMSNorms, visual tower (27 blocks / 333 weights), and MTP head tensors | Native BF16 | 348 | Protects vocabulary tail, multi-modal reasoning, and MTP draft survival |
| Tier 2 (Medium) | Self-Attention (q_proj, k_proj, v_proj, o_proj), GDN Recurrence () (Layers 2–61), and Boundary MLPs (Layers 0, 1, 62, 63) |
Standard GPTQ assigns a single static exponent code per group block (amax → code 6). Kearuga's 4o6 algorithm evaluates dynamic range block-by-block and selects the optimal scale:
In our calibration, 44.7% of blocks selected Code 4, yielding a 16% reduction in Hessian-weighted MSE (0.529 → 0.445) and cutting held-out KL divergence by 21.7% without increasing tensor bit-width or memory footprint.
SGLang's high-performance Blackwell kernels fuse gate_proj and up_proj into a single GEMM operation, indexing a unified weight_scale_2.max(). Quantizing gate and up independently introduces scale divergence that breaks dequantization. Kearuga enforces an exact fused-shared global scale across every layer:
text
scale_shared = max(amax(gate), amax(up))This ensures bit-exact kernel compatibility and eliminates dequantization distortion.
Kearuga is served directly via official SGLang Docker container images (sha256:616a3e97… or later) with native DFlash 2 speculative acceleration. The recommended drafter is 0xWhiteMage/Qwen3.8-27B-Kearuga-DFlash2 v1.0 (Kearuga-distilled, NVFP4, K=12):
bash
docker run --gpus all \ --shm-size 32g \ --ulimit memlock=-1:-1 \ --cap-add IPC_LOCK \ -p 8888:8888 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ lmsysorg/sglang@sha256:616a3e97f45191af975896cfa644279096cb31bd408a071c2e99ca7209c3cafe \ python3 -m sglang.launch_server \ --model-path 0xWhiteMage/Qwen3.8-27B-Kearuga \ --served-model-name "Qwen3.8 27B Kearuga" \ --speculative-algorithm DFLASH \ --speculative-draft-model-path 0xWhiteMage/Qwen3.8-27B-Kearuga-DFlash2 \ --speculative-draft-model-quantization modelopt_fp4 \ --speculative-num-draft-tokens 12 \ --speculative-draft-window-size 2048 \ --kv-cache-dtype bf16 \ --mem-fraction-static 0.85 \ --max-total-tokens 1048576 \ --context-length 262144 \ --max-running-requests 4 \ --host 0.0.0.0 \ --port 8888The 64K draft-head overlay adds a further ≈ 6 % on top of plain SGLang (35.33 vs 33.30 tok/s C1 aggregate) — it is documented on the drafter card and is handled automatically by the serving repo's launcher ./start-dflash2.sh.
Alternative: stock drafter (K=10) — the serving repo's DRAFTER_PROFILE=stock (the original v0.5.0 recipe) with z-lab/Qwen3.8-27B-DFlash2: use the command above with --speculative-draft-model-path z-lab/Qwen3.8-27B-DFlash2, no --speculative-draft-model-quantization, and --speculative-num-draft-tokens 10.
| Parameter | Recommended Setting | Rationale |
|---|---|---|
--speculative-num-draft-tokens | 12 with the Kearuga drafter | K=8 loses code/math acceptance 7–12 %; K=14/16 give nothing at C1 and lose 2–6 % at C4. 10 with the stock drafter. |
--kv-cache-dtype | bf16 | Preserves full logit fidelity (KL: 0.0170 → 0.0165, exact continuations: 20/40 vs 19/40) with negligible compute cost. |
--context-length | 262144 |
In multi-agent production pipelines, background batch traffic can cause interactive request freezes. Kearuga's serving stack enables native priority preemption:
json
{ "model": "Qwen3.8 27B Kearuga", "priority": 100, "messages": [ {"role": "user", "content": "Analyze this mission-critical trace..."} ]}| Server Load State | Default Priority TTFT | Interactive Priority ("priority": 100) | Latency Reduction |
|---|---|---|---|
| DFlash 2 (All 4 Seats Saturated) | ~43.15 s | ~2.63 s | 93.9% faster |
Every promoted build of Kearuga must satisfy a strict 15-gate fail-closed verification harness prior to promotion:
gate_proj + up_proj weight_scale_2 are identical across all layers.19 × 23 = 437) and logic traps (9.9 vs 9.11).| File Name | File Size | Description & Scope |
|---|---|---|
config.json | 8.2 KB | Architectural model hyperparameters & quantization configuration |
generation_config.json | 240 B | Sampling defaults, EOS tokens, and generation bounds |
hf_quant_config.json | 14.1 KB | Detailed ModelOpt mixed-precision tier mappings & scale rules |
model.safetensors.index.json | 185 KB |
0xWhiteMage/Qwen3.8-27B-Kearuga-DFlash2 — Kearuga-distilled, Kearuga-calibrated NVFP4 drafter + 64K draft head (served by serving stack v0.6.6)z-lab/Qwen3.8-27B-DFlash2Qwen/Qwen3.8-27BIf you use Kearuga or build upon its tiered sensitivity architecture in your work, please cite:
bibtex
@misc{kearuga-2026, title={Kearuga: Hybrid GPTQ-4o6 + FP8 Quantization of Qwen3.8-27B for Speculative-Decoding Serve on NVIDIA DGX Spark}, author={0xWhiteMage}, year={2026}, publisher={Hugging Face}, url={https://huggingface.co/0xWhiteMage/Qwen3.8-27B-Kearuga}}This checkpoint is released under the Apache 2.0 License. The underlying base model Qwen/Qwen3.8-27B is subject to its original license terms from Alibaba Cloud.
{ "model": "Qwen3.8 27B Kearuga", "priority": 100, "messages": [ {"role": "user", "content": "Analyze this mission-critical trace..."} ]}| Virtually identical distribution profile |
| Fidelity-40 Top-1 Agreement | 40 / 40 | 40 / 40 (100%) | Zero argmax token divergence across 40 evaluation prompts |
| Fidelity-40 Exact 32-Token Continuation | 40 / 40 | 20 / 40 (50.0%) | 50% byte-identical; remaining 50% diverge only at near-tie logits |
| Held-Out Full-Vocab KL (72,082 tokens) | 0.0000 | 0.0208 | Verified on held-out text sequences |
| Held-Out Top-1 Agreement | 100.0% | 95.0% | Exceptional semantic adherence across broad corpora |
| Quality-200 Objective Score | — | 157 / 180 | GSM8K: 66/80 · HumanEval: 39/40 · IFEval: 34/40 · Agentic: 18/20 — 155/180 served with the Kearuga drafter (near-tie band 154–159) |
Decode throughput (Kearuga drafter, bench/scale.py) | — | C1 44.7 · C2 78–94 · C4 131 tok/s aggregate | T=0, thinking off, 512 forced tokens, aggregate incl. TTFT (0.23 / 0.28 / 0.33 s); measured 2026-09-17, identical with the GPU clock lock on or off (the decode loop is memory-bandwidth-bound) |
| Decode throughput without speculation (same target, no drafter, serving-repo 50-prompt battery) | — | C1 10.30 · C4 39.56 tok/s aggregate (flat ≈ 10.3 tok/s per stream) | Thinking off, T=0, single tenant, 2026-09-23; the production configuration (Kearuga drafter K=12, 64K head, FP8 draft KV) measures 35.66 / 108.83 on the same battery → the drafter is worth 3.46× C1 / 2.75× C4 (serving stack v0.6.7) |
| Serving footprint at --mem-fraction-static 0.85 | — | ~100.6 GB static (24.9 GB weights + 56.9 GB BF16 target KV + 8.9 GB FP8 drafter KV for 932,355 tokens + 5.8 GB GDN state + drafter + graphs) | Measured at the v0.6.6 production boot (2026-09-22); 4 seats × 262K window sharing a 932K-token pool (818,294 tokens with BF16 drafter KV, v0.6.5 and earlier) |
in_proj| FP8 E4M3 (Block / Static scales) |
| 205 (195 + 10) |
Keeps DFlash 2 tap layers [5, 19, 33, 47, 61] pristine; stabilizes recurrences |
| Tier 3 (Core) | MLP gate_proj & up_proj (Layers 2–61, 60 layers × 2) | W4A16 NVFP4 (GPTQ-4o6) | 120 | GPTQ with Four-Over-Six group scaling eliminates 21.7% of weight-state KL divergence |
| Tier 4 (Down) | MLP down_proj (Layers 2–61, 60 layers × 1) | NVFP4 AWQ (Pre-quantized) | 60 | Preserves outlier scales via calibrated channel activation protection |
| Full native context supported without YaRN interpolation. |
--mem-fraction-static | 0.85 + --max-total-tokens 1048576 | The validated production pair: at 0.85 the pool comes out at 932,355 tokens with FP8 drafter KV (serving stack v0.6.6; 818,294 with BF16 drafter KV) — the cap is a ceiling, not the allocation — leaving ≈ 15 GB GPU-side headroom for CUDA graphs and the OS. |
| Complete tensor index mapping all 2,009 tensors to shards |
model-00001-of-00003.safetensors | 9.96 GB | Shard 1 (layers 0–21 weights & attention projections) |
model-00002-of-00003.safetensors | 9.99 GB | Shard 2 (layers 22–43 weights & attention projections) |
model-00003-of-00003.safetensors | 4.94 GB | Shard 3 (layers 44–63 weights & lm_head) |
model-mtp.safetensors | 0.85 GB | Native Multi-Token Prediction (MTP) draft head (BF16) |
tokenizer.json | 11.4 MB | Full BPE vocabulary (151,646 tokens) |
tokenizer_config.json | 7.3 KB | Chat templates, special tokens, and tokenizer definitions |