kirillbilchenko

Qwen3.8-27B-MxWave-MXFP4-FP8

README

License: apache-2.0

Quick start

Download with the current Hugging Face CLI:

bash
HF_XET_HIGH_PERFORMANCE=1 hf download \
  kirillbilchenko/Qwen3.8-27B-MxWave-MXFP4-FP8 \
  --local-dir ./Qwen3.8-27B-MxWave-MXFP4-FP8

The tested runtime is official vLLM 0.29.0 on DGX Spark / SM121:

bash
vllm serve ./Qwen3.8-27B-MxWave-MXFP4-FP8 \
  --load-format safetensors \
  --dtype bfloat16 \
  --linear-backend marlin \
  --max-model-len 16384 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.45 \
  --kv-cache-dtype bfloat16 \
  --no-enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192

The FP8 and MXFP4 kernels are weight-only on this tested path; activations remain BF16. Do not use the obsolete VLLM_TEST_FORCE_FP8_MARLIN compatibility switch with vLLM 0.29.0.

Precision allocation

The candidate promotes these runtime-safe groups:

sequence-output projections in layers 8-11: three linear-attention out_proj tensors and one full-attention o_proj tensor;
MLP input projections in layers 51-54: four gate_proj and four up_proj tensors.

The remaining 388 targeted language projections retain the original H64 OCP MXFP4 E2M1 weights and E8M0 block-32 scales. Embeddings, language head, norms, MTP tensors, vision tensors, and other non-target tensors remain source precision. The exact allocation is recorded in mxwave-manifest.json.

Selection method

The allocation was fixed before final evaluation:

MxWave generated 12 semantic candidates spanning four runtime-operation families and three depth bands.
Each candidate was compared with H64 against the same BF16 next-token distributions on two disjoint 16-context splits.
A candidate was eligible only when mean forward KL improved on both splits.
Selection admitted at most four buckets and 1 GiB of added tensor data.
The combined allocation then had to improve both halves of an untouched 96-context holdout without losing more than one BF16 top-1 agreement.

The final allocation promotes 12 of 400 quantized tensors and adds 393,854,976 tensor bytes (375.6 MiB). It does not use task labels for tensor selection.

The implementation and model-independent procedure are in MxWave and its precision-budget runbook.

Evaluation

All H64/candidate comparisons below are paired: same prompts, task definitions, seeds, decoding, runtime image, and backend. The final runtime was vllm/vllm-openai:v0.29.0 at digest sha256:c2914767605584b6d8f45686b82de173ecc99e781897aa3d0a66dacd72c51ae1.

WikiText-2 prompt perplexity

The primary deterministic evaluation used 316 independent 4,096-character windows and 297,199 scored tokens.

Table with columns: Model, Prompt PPL, Candidate comparison
Model	Prompt PPL	Candidate comparison
BF16 source	7.980864	1.6378% higher PPL
MxWave H64	8.119445	0.0969% lower PPL
MxWave MXFP4/FP8	8.111578	baseline
AMD Quark-AWQ MXFP4	8.187544	0.9278% lower PPL

The paired candidate-versus-H64 95% bootstrap interval was [-0.1297%, -0.0637%], entirely below zero. Absolute perplexities are specific to this character-window protocol and must not be compared directly with scores from different token windows or strides.

Untouched distribution holdout

Table with columns: Metric, H64, Candidate
Metric	H64	Candidate
Mean forward KL to BF16	0.047117	0.044942
BF16 top-1 agreement	89/96	89/96

Candidate-minus-H64 mean forward KL was -0.002176 nats and was negative on both predeclared holdout halves. The paired interval [-0.005939, +0.000523] crossed zero, so this holdout passed the directional gate but is not an independent statistically conclusive claim.

Full deterministic GSM8K

This run used lm-evaluation-harness 0.4.13, all 1,319 test examples, five-shot prompts, greedy decoding, temperature zero, and 1,024 maximum generated tokens. The frozen gsm8k_nothink task SHA-256 was 0a5c248412fa400d32386b4c8c65aeb025711650dda0e6c0ec39e67e86cda5c7; both runs completed with zero failed requests.

Table with columns: Metric, H64, Candidate, Difference
Metric	H64	Candidate	Difference
Strict match	1,208/1,319 (91.5845%)	1,214/1,319 (92.0394%)	+0.4549 points
Flexible extract	1,208/1,319 (91.5845%)	1,215/1,319 (92.1152%)	+0.5307 points

For strict matching, candidate-only wins were 25 and H64-only wins were 19. Exact McNemar was p=0.4514; the paired interval was [-0.5307, +1.4405] points. The direction is favorable and passes the predeclared one-point non-inferiority gate, but it is not a statistically proven GSM8K improvement.

Serving tradeoff

The fixed workload used random 512-token inputs, 128 generated tokens, temperature zero, ignored EOS, and disabled prefix caching.

Table with columns: Model, Concurrency, Output tok/s
Model	Concurrency	Output tok/s
H64	1	12.484
Candidate	1	12.213
H64	4	42.193
Candidate	4	41.230-41.250

The candidate was 2.17% slower at concurrency 1 and 2.24-2.28% slower at concurrency 4, within the frozen 3% budget. The deterministic JSON, tool-use, short-completion, and Python smoke suite passed all 12 repeated checks.

Machine-readable summaries:

Limitations

The measured improvements are small. This is a Pareto option, not a claim that every user should prefer it over H64.
GSM8K favored the candidate but its paired uncertainty includes zero.
Evaluation was text-only. Vision tensors are present and unchanged, but image capability was not independently evaluated.
Long-context retrieval, coding benchmarks, and broad instruction-following suites were not used to make the claims above.
Runtime support depends on vLLM, compressed-tensors, CUDA, and accelerator versions. The checkpoint is not a GGUF, MLX, llama.cpp, Ollama, or LM Studio model.
The selected allocation is specific to Qwen3.8-27B. The method is architecture-adapter based, but another model must run its own frozen screen.

Reproducibility and provenance

Source model: Qwen/Qwen3.8-27B revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0.
Public MxWave precision-budget implementation: 6f7ab4ec.
H64 parent manifest SHA-256: 8c9e4205941263bec879712fc790aabd8f75cc23dce53d9fb13daecb2161040f.
Mixed config SHA-256: ff5cbabc9c487eea826373b209492e1b5e3d3eaa4b4216729cfe9f4fa4c81018.
Mixed index SHA-256: d359434efbacc1a93c38221d5d554883d27554c2a77e55b41804ac09735db922.
Mixed manifest SHA-256: ed3ed368221813a1c5bc9ab7dcc150506e2d254172446f1f493bba6923983263.
Ordered 18-shard hash-of-hashes: .

The detailed selection protocol, uncertainty, hashes, and runtime results are in the Qwen precision-budget record.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

kirillbilchenko

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality