Qwen3.8-27B-nvfp4-v17 API & Inference Endpoint

Model Details

Qwen3.8-27B (qwen3_5 / Qwen3_5ForConditionalGeneration):

Type: Vision-language model (hybrid linear + full attention language model + vision encoder)
Parameters: 27B (language model) + 460.7M (vision tower)
Layers: 64 — 48 linear-attention (GatedDeltaNet) + 16 full-attention (every 4th layer: 0, 4, 8, …, 60)
Hidden size: 5,120
Intermediate size: 17,408
Attention heads: 24 × 256 dim
Vocabulary: 248,320
Activation: SiLU
Context: 8,192 tokens (benchmark configuration)
Vision encoder: 27-layer ViT, hidden size 1,152, patch size 16, 16 heads

Each linear-attention layer has in_proj_qkv, in_proj_z, out_proj plus recurrent state paths in_proj_a/in_proj_b. Each full-attention layer has standard q/k/v/o_proj. All layers share gate_proj/up_proj/down_proj MLP blocks. The vision tower (model.visual.*) processes image/video patches and injects embeddings into the language model via image/video token boundaries.

Quantization Configuration

Precision Assignment

Table with columns: Module group, Format, Weights, Activations, Layers
Module group	Format	Weights	Activations	Layers
MLP (gate/up/down)	FP8 W8A8 (E4M3)	8-bit float, sym, `channel`, `memoryless_minmax`, `actorder=static`	8-bit float, sym, `token`, dynamic	all 64
self_attn (Q/K/V/O)	FP8 W8A8 (E4M3)	same	same	all 16 full-attn layers

GPTQ: Hessian-based weight correction, sequential targets Qwen3_5DecoderLayer, actorder=static, dampening_frac=0.01 (default).

Modules Kept in BF16

Table with columns: Pattern, Reason
Pattern	Reason
`visual.*`	Vision tower — small, quantization-sensitive
`linear_attn.norm`	Numerically fragile normalization path
`linear_attn.in_proj_a`	Non-power-of-64 dims — CUTLASS constraint
`linear_attn.in_proj_b`	Non-power-of-64 dims — CUTLASS constraint
`mtp.*`	Multi-token-prediction head — kept BF16; functional for MTP speculative decoding (accept length ~1.70–1.77 after 2026-08-17 config fix)

Calibration

Source: malaiwah/qwen38-27b-fidelity-suite-v3 tokens (181 contexts × 2048 tokens)
Samples: 181
Max sequence length: 2048
Format: Decoded token IDs from suite, not raw text

Usage

bash
# vLLM serve (requires Blackwell SM120 + FlashInfer)
vllm serve /models/Qwen3.8-27B-nvfp4-gptq-v17-vision \
  --max-model-len 8192 \
  --max-num-seqs 512 \
  --quantization compressed-tensors \
  --trust-remote-code

--max-num-seqs matters on this architecture. 48 of 64 layers use linear attention, and vLLM allocates one Mamba-style cache block per decode sequence. The default max_num_seqs=1024 can exceed available blocks and fail during CUDA graph capture with max_num_seqs (1024) exceeds available Mamba cache blocks. Lower to 512 or raise --gpu-memory-utilization. This is a property of the base model, not of quantization.

python
from vllm import LLM, SamplingParams

llm = LLM(model="/models/Qwen3.8-27B-nvfp4-gptq-v17-vision",
          quantization="compressed-tensors",
          max_model_len=8192,
          trust_remote_code=True)

# Text-only generation (identical to text-only v17)
out = llm.generate(
    ["Explain 4-bit quantization in two sentences."],
    SamplingParams(temperature=0.7, max_tokens=256),
)
print(out[0].outputs[0].text)

# Vision: image + text via chat template
from vllm.inputs import TextPrompt
messages = [{
    "role": "user",
    "content": [
        {"type": "image_url", "image_url": {"url": "https://example.com/image.png"}},
        {"type": "text", "text": "Describe this image."},
    ],
}]
prompt = llm.get_tokenizer().apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True)
out = llm.generate(
    {"prompt": prompt, "multi_modal_data": {"image": image_placeholder}},
    SamplingParams(temperature=0.7, max_tokens=512),
)
print(out[0].outputs[0].text)

Docker Compose

A docker-compose.yml is provided for one-command deployment. The image bundles the vLLM infernal-invocation fork with FlashInfer and SM120 kernels; only the quantized checkpoint must be mounted from the host.

bash
# Place the checkpoint at ./models/Qwen3.8-27B-nvfp4-gptq-v17-vision, then:
docker compose up -d

# Or override the model path and port:
MODEL_DIR=/data/models/Qwen3.8-27B-nvfp4-gptq-v17-vision PORT=8001 docker compose up -d

The endpoint is OpenAI-compatible: http://localhost:8000/v1.

yaml
# docker-compose.yml — Qwen3.8-27B-FP8-GPTQ-v17
#
# Serves the v17 all-FP8 W8A8 GPTQ quantization via vLLM with FlashInfer.
# Image: lribeiro/qwen38-quant:ii-r15-p1  (vLLM infernal-invocation fork, SM120)
#
# Requirements:
#   - NVIDIA Blackwell (SM120) GPU with FP8 E4M3 tensor cores
#   - NVIDIA Container Toolkit (nvidia-docker) installed on host
#   - Quantized checkpoint at ${MODEL_DIR} (default: ./models/Qwen3.8-27B-nvfp4-gptq-v17)
#
# Customize via .env file or shell environment variables:
#
#   PORT=8000                           API server port
#   SERVED_MODEL_NAME=qwen38-27b        Model name returned by /v1/models
#   MAX_MODEL_LEN=8192                  Maximum context length (tokens)
#   MAX_NUM_SEQS=512                    Maximum concurrent decode sequences
#   MAX_NUM_BATCHED_TOKENS=8192         Maximum tokens per batching iteration
#   GPU_MEMORY_UTILIZATION=0.90         Fraction of GPU memory for vLLM (0.0–1.0)
#   TENSOR_PARALLEL_SIZE=1              Tensor-parallel degree (1, 2, 4)
#   KV_CACHE_DTYPE=fp8                  KV cache dtype: fp8 | auto | bfloat16
#   CUDA_VISIBLE_DEVICES=0              GPU index(s) to use
#   MODEL_DIR=./models/Qwen3.8-27B-nvfp4-gptq-v17   Host path to checkpoint

services:
  qwen38-27b-v17:
    image: lribeiro/qwen38-quant:ii-r15-p1 # ii with mxfp4 patches and other minor corrections
    container_name: qwen38-27b-v17
    ipc: host
    ports:
      - "${PORT:-8000}:${PORT:-8000}"
    volumes:
      # Mount the quantized model checkpoint (read-only)
      - ${MODEL_DIR:-./models/Qwen3.8-27B-nvfp4-gptq-v17}:/models/Qwen3.8-27B-nvfp4-gptq-v17:ro
      # Persistent JIT / compile caches (avoids recompilation on restart)
      - vllm-cache:/cache
      # HuggingFace hub cache (tokenizer / config downloads)
      - hf-cache:/data/hf_cache
    environment:
      # ── GPU / runtime ──
      - CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-0}
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
      - SAFETENSORS_FAST_GPU=1
      - OMP_NUM_THREADS=16
      # ── vLLM engine ──
      - VLLM_USE_AOT_COMPILE=1
      - VLLM_USE_FLASHINFER_SAMPLER=1
      - VLLM_USE_V2_MODEL_RUNNER=1
      - VLLM_ALLOW_INSECURE_SERIALIZATION=1
      # SM120: disable broken FP4/FP8 kernels — use FlashInfer / b12x /  CUTLASS paths
      - VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel,MarlinFP8ScaledMMLinearKernel
      # ── Cache directories ──
      - HF_HOME=/data/hf_cache
      - VLLM_CACHE_DIR=/cache/vllm
      - TORCHINDUCTOR_CACHE_DIR=/cache/torchinductor
      - TRITON_CACHE_DIR=/cache/triton
      - FLASHINFER_WORKSPACE_BASE=/cache/flashinfer
    entrypoint: ["python", "-m", "vllm.entrypoints.cli.main"]
    command:
      - serve
      - /models/Qwen3.8-27B-nvfp4-gptq-v17
      - --served-model-name=${SERVED_MODEL_NAME:-qwen38-27b}
      - --host=0.0.0.0
      - --port=${PORT:-8000}
      - --trust-remote-code
      - --quantization=compressed-tensors
      - --tensor-parallel-size=${TENSOR_PARALLEL_SIZE:-1}
      - --kv-cache-dtype=${KV_CACHE_DTYPE:-fp8}
      - --block-size=128
      - --load-format=fastsafetensors
      - --gpu-memory-utilization=${GPU_MEMORY_UTILIZATION:-0.94}
      - --max-model-len=${MAX_MODEL_LEN:-262144}
      - --max-num-seqs=${MAX_NUM_SEQS:-512}
      - --max-num-batched-tokens=${MAX_NUM_BATCHED_TOKENS:-8192}
      - --generation-config=vllm
      - --reasoning-parser=qwen3
      - --attention-backend=flashinfer
      - --linear-backend=auto
      - --max-cudagraph-capture-size=16
      - --async-scheduling
      - --enable-chunked-prefill
      - --enable-prefix-caching
      - --enable-flashinfer-autotune
      - >-
        --compilation-config={"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]

volumes:
  vllm-cache:
  hf-cache:

Configurable variables (set in .env or shell):

Table with columns: Variable, Default, Description
Variable	Default	Description
`PORT`	`8000`	API server port (OpenAI-compatible)
`SERVED_MODEL_NAME`	`qwen38-27b`	Model name returned by `/v1/models`
`MAX_MODEL_LEN`	`262144`	Maximum context length in tokens

Evaluation

Distribution Fidelity (KLD)

Hidden-state capture after final RMSNorm → shared BF16 LM-head replay → full-vocabulary KL(BF16_ref ‖ candidate), two-pass log-sum-exp normalization. Body-only comparison (candidate LM head unused). Suite: 136 contexts × 2048 tokens, 278,392 scored positions over 248,320-token vocabulary.

Table with columns: Metric, Value
Metric	Value
Token mean KLD	0.012310
Token median KLD	0.002351
p95 KLD	0.047520
p99 KLD	0.191077
p99.9 KLD	0.694279
Max KLD	3.961384
Mean JSD (bits)	0.004245
Top-1 agreement	96.17%
Validation tier

KLD by Domain

Table with columns: Stratum, Contexts, Mean KLD
Stratum	Contexts	Mean KLD
scientific	39	0.00409
multilingual	7	0.00439
encyclopedic	13	0.00866
code	36	0.01276
literary	41	0.02224

Literary text is the dominant KLD contributor — 5.4× the scientific stratum. FP8 quantization affects creative/prose generation more than technical text, but the gap is narrower than in 4-bit configs (v6: 4.9×, v7: 4.9× — v17 has a wider but lower-magnitude spread due to its lower baseline KLD).

Throughput

Single NVIDIA RTX PRO 6000 Blackwell (SM120, 96 GB), vLLM 0.26.1rc0, FlashInfer backend. Duration-based sustained decode, 30s cells, temperature 0.0, ignore_eos=true.

Table with columns: Metric, Value
Metric	Value
Prefill (2k context)	7,306 tok/s
TTFT (2k)	0.279 s
Decode C1 (0 ctx)	49.8 tok/s
Decode C1 (2k ctx)	49.4 tok/s
Decode C4 (0 ctx)	180.6 tok/s
Decode C4 (2k ctx)	176.1 tok/s

Composite Score

$score = KLD \times 106 prefill 2 k \times decode c 1 =$

Context Within Experiment Series

v17 is the reference all-FP8 W8A8 baseline (Phase 2). It established that uniform FP8 with GPTQ correction strictly dominates the entire NVFP4 W4A4 line: lower KLD than the best 4-bit config (v6: 0.0160) at comparable speed, with much higher precision (E4M3 8-bit vs E2M1 4-bit). Three follow-up experiments confirmed v17's hyperparameters are optimal:

Table with columns: Experiment, Change vs v17, KLD, vs v17, Score
Experiment	Change vs v17	KLD	vs v17	Score
v17 (baseline)	—	0.01231	—	29.55
v23	damp=0.001	0.01241	+1%	29.03
v24	damp=0.005, block256	0.01253	+2%	28.52

GPTQ dampening is irrelevant for FP8. Tested 0.001–0.1 — all produce 0.0124 ± noise. The GPTQ Hessian is well-conditioned for FP8 regardless of dampening_frac.

Calibration sample count doesn't matter. v16 (512 samples) vs v17 (181 samples) — KLD 0.01245 vs 0.01231, within noise. The Hessian is well-estimated at 181 samples.

Phase 3 (v18–v20, v27–v30) used v17 as the baseline for KLD attribution by layer type, finding that MLP FP8 contributes ~0.009 KLD (dominant), linear_attn activation quantization ~0.0025, and self_attn ~0.0006. Phase 4 (v30–v34) built on this to produce v31 (W8A16 weight-only attention), the overall best model at KLD 0.01014, score 31.45.

Pareto Position

Table with columns: Objective, Best Model, KLD, Prefill, Decode, Score
Objective	Best Model	KLD	Prefill	Decode	Score
Best composite score	v31	0.01014	6,385	49.9	31.45
Best all-FP8 (uniform)	v17	0.01231	7,306	49.8	29.55
Lowest KLD (our quants)

Provenance & Integrity

Table with columns: Artifact, SHA-256
Artifact	SHA-256
Model index	`aa62fb91085564f542299f80c1447e044fca41fbb89350d2979adc0e2804e097`
Config	`dc8f1def22f7c0f5ed182444c8431d16dbdc9aab398c894918c864e96c78ad74`
Suite tokens	`3f9d17f1b55f64872ad3ac19c8711654e09ba70b7ca14b0851525088fe735691`
Capture manifest	`fb2317dd353c01b4544427ded1c2966b4276e370ae2d1b492ac052858ce9d997`
Shared LM head	`25a30fd5f826da0abc4efc4cc71def9f02bcb8085f7175eee284d221dee4cfff`

14 model shards individually hashed in reports/report-nvfp4-gptq-v17.json → candidate_identity.shard_sha256. Full fidelity report, benchmark JSON, and hidden-state captures archived in the experiment workspace.

Hardware & Runtime

Table with columns: Field, Value
Field	Value
GPU	4× NVIDIA RTX PRO 6000 Blackwell Max-Q (SM120, 96 GiB GDDR7 each)
Driver	595.58.03
Benchmark GPU count	1
VRAM used	86,606 / 97,887 MB (88.5%)
Temp (avg / max)	53.0 °C / 59.0 °C

Reproduction

The language-model weights are byte-identical to the original text-only v17. This checkpoint was produced by merging the existing v17 quantized shards with the BF16 visual tower + MTP head extracted from the base model — no re-quantization needed.

Step 1: Original v17 quantization (already done)

bash
# Inside Docker (vLLM infernal-invocation image, FlashInfer, SM120)
CUDA_VISIBLE_DEVICES=0 python /data/scripts/quantize_nvfp4_gptq_v17.py \
  --model /data/models/Qwen3.8-27B-bf16 \
  --output /data/models/Qwen3.8-27B-nvfp4-gptq-v17 \
  --calib-samples 181 \
  --calib-max-len 2048

Calibration tokens from $SUITE_DIR/tokens (default /data/suite-v3/tokens). Script: scripts/quantize_nvfp4_gptq_v17.py. Log: quant_v17.log.

Step 2: Merge visual tower + MTP from base (this checkpoint)

bash
# Merge v17 quantized LM shards with BF16 visual+mtp from base model
python /workspace/scripts/merge_v17_vision.py \
  --v17 /models/Qwen3.8-27B-nvfp4-gptq-v17 \
  --base /models/Qwen3.8-27B-bf16 \
  --output /data/models/Qwen3.8-27B-nvfp4-gptq-v17-vision

Script: scripts/merge_v17_vision.py. Extracts 333 visual + 15 MTP tensors from the BF16 base, writes them as a 15th shard, copies the 14 v17 shards unchanged, and patches config.json to Qwen3_5ForConditionalGeneration (VLM) with vision_config.

Note: The v17 script's default --calib-samples is 1024, but the suite contains only 181 contexts. llm-compressor warns
markdown
Requested 1024 samples but the provided dataset only has 181 samples
and uses all 181. The effective calibration set is 181 samples, matching v6/v16/v31. Pass --calib-samples 181 to suppress the warning.

Limitations

KLD far from malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated
Superseded by v31. v31 (FP8 W8A8 MLP + W8A16 weight-only attention) achieves 18% lower KLD (0.01014 vs 0.01231) at similar decode speed. Use v17 when uniform FP8 is required or when the faster prefill (7,306 vs 6,385 tok/s) matters more than the KLD gap.
Attention activation quantization is the second-largest KLD contributor. FP8 W8A8 quantizes attention activations; v31 eliminates this by keeping BF16 activations for attention (W8A16 weight-only). KLD attribution: MLP ~0.009, linear_attn ~0.0025, self_attn ~0.0006.
Blackwell-optimized. FP8 E4M3 MMA is supported on SM120 (Blackwell), SM89/SM90 (Ada/Hopper), but the benchmark and KLD capture were run on SM120 only. Validate on your hardware.
Body-only KLD. Fidelity measured against a shared BF16 LM head; end-to-end generation quality may differ slightly.
Literary domain degradation. 5.4× higher KLD on literary text vs scientific — creative/prose generation is disproportionately affected, though less severely than in 4-bit configs.
Quantization is lossy. Validate on your own workload before production use.

Acknowledgements

Standing on the shoulders of giants. This work would not exist without the rtx6kpro community and the broader local-inference-lab Discord — the open protocol, tooling, and baseline measurements that made systematic quantization comparison possible.

The distribution-fidelity (KLD) methodology follows the published protocol from Kimi-K3 distribution fidelity 1024×2048. The reference harness (glm52_exl3_shared_h_kld.py) and the benchmark tool (llm-inference-bench) are adapted from the rtx6kpro repository. The Gilded Gnosis EXL3 model cards established the per-bit error-ladder and validation-tier framework that this sweep builds on.

Model Details

Qwen3.8-27B (qwen3_5 / Qwen3_5ForConditionalGeneration):

Type: Vision-language model (hybrid linear + full attention language model + vision encoder)
Parameters: 27B (language model) + 460.7M (vision tower)
Layers: 64 — 48 linear-attention (GatedDeltaNet) + 16 full-attention (every 4th layer: 0, 4, 8, …, 60)
Hidden size: 5,120
Intermediate size: 17,408
Attention heads: 24 × 256 dim
Vocabulary: 248,320
Activation: SiLU
Context: 8,192 tokens (benchmark configuration)
Vision encoder: 27-layer ViT, hidden size 1,152, patch size 16, 16 heads

Quantization Configuration

Precision Assignment

Table with columns: Module group, Format, Weights, Activations, Layers
Module group	Format	Weights	Activations	Layers
MLP (gate/up/down)	FP8 W8A8 (E4M3)	8-bit float, sym, `channel`, `memoryless_minmax`, `actorder=static`	8-bit float, sym, `token`, dynamic	all 64
self_attn (Q/K/V/O)	FP8 W8A8 (E4M3)	same	same	all 16 full-attn layers

GPTQ: Hessian-based weight correction, sequential targets Qwen3_5DecoderLayer, actorder=static, dampening_frac=0.01 (default).

Modules Kept in BF16

Table with columns: Pattern, Reason
Pattern	Reason
`visual.*`	Vision tower — small, quantization-sensitive
`linear_attn.norm`	Numerically fragile normalization path
`linear_attn.in_proj_a`	Non-power-of-64 dims — CUTLASS constraint
`linear_attn.in_proj_b`	Non-power-of-64 dims — CUTLASS constraint
`mtp.*`	Multi-token-prediction head — kept BF16; functional for MTP speculative decoding (accept length ~1.70–1.77 after 2026-08-17 config fix)

Calibration

Source: malaiwah/qwen38-27b-fidelity-suite-v3 tokens (181 contexts × 2048 tokens)
Samples: 181
Max sequence length: 2048
Format: Decoded token IDs from suite, not raw text

Usage

bash
# vLLM serve (requires Blackwell SM120 + FlashInfer)
vllm serve /models/Qwen3.8-27B-nvfp4-gptq-v17-vision \
  --max-model-len 8192 \
  --max-num-seqs 512 \
  --quantization compressed-tensors \
  --trust-remote-code

--max-num-seqs matters on this architecture. 48 of 64 layers use linear attention, and vLLM allocates one Mamba-style cache block per decode sequence. The default max_num_seqs=1024 can exceed available blocks and fail during CUDA graph capture with max_num_seqs (1024) exceeds available Mamba cache blocks. Lower to 512 or raise --gpu-memory-utilization. This is a property of the base model, not of quantization.

python
from vllm import LLM, SamplingParams

llm = LLM(model="/models/Qwen3.8-27B-nvfp4-gptq-v17-vision",
          quantization="compressed-tensors",
          max_model_len=8192,
          trust_remote_code=True)

# Text-only generation (identical to text-only v17)
out = llm.generate(
    ["Explain 4-bit quantization in two sentences."],
    SamplingParams(temperature=0.7, max_tokens=256),
)
print(out[0].outputs[0].text)

# Vision: image + text via chat template
from vllm.inputs import TextPrompt
messages = [{
    "role": "user",
    "content": [
        {"type": "image_url", "image_url": {"url": "https://example.com/image.png"}},
        {"type": "text", "text": "Describe this image."},
    ],
}]
prompt = llm.get_tokenizer().apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True)
out = llm.generate(
    {"prompt": prompt, "multi_modal_data": {"image": image_placeholder}},
    SamplingParams(temperature=0.7, max_tokens=512),
)
print(out[0].outputs[0].text)

Docker Compose

bash
# Place the checkpoint at ./models/Qwen3.8-27B-nvfp4-gptq-v17-vision, then:
docker compose up -d

# Or override the model path and port:
MODEL_DIR=/data/models/Qwen3.8-27B-nvfp4-gptq-v17-vision PORT=8001 docker compose up -d

The endpoint is OpenAI-compatible: http://localhost:8000/v1.

yaml
# docker-compose.yml — Qwen3.8-27B-FP8-GPTQ-v17
#
# Serves the v17 all-FP8 W8A8 GPTQ quantization via vLLM with FlashInfer.
# Image: lribeiro/qwen38-quant:ii-r15-p1  (vLLM infernal-invocation fork, SM120)
#
# Requirements:
#   - NVIDIA Blackwell (SM120) GPU with FP8 E4M3 tensor cores
#   - NVIDIA Container Toolkit (nvidia-docker) installed on host
#   - Quantized checkpoint at ${MODEL_DIR} (default: ./models/Qwen3.8-27B-nvfp4-gptq-v17)
#
# Customize via .env file or shell environment variables:
#
#   PORT=8000                           API server port
#   SERVED_MODEL_NAME=qwen38-27b        Model name returned by /v1/models
#   MAX_MODEL_LEN=8192                  Maximum context length (tokens)
#   MAX_NUM_SEQS=512                    Maximum concurrent decode sequences
#   MAX_NUM_BATCHED_TOKENS=8192         Maximum tokens per batching iteration
#   GPU_MEMORY_UTILIZATION=0.90         Fraction of GPU memory for vLLM (0.0–1.0)
#   TENSOR_PARALLEL_SIZE=1              Tensor-parallel degree (1, 2, 4)
#   KV_CACHE_DTYPE=fp8                  KV cache dtype: fp8 | auto | bfloat16
#   CUDA_VISIBLE_DEVICES=0              GPU index(s) to use
#   MODEL_DIR=./models/Qwen3.8-27B-nvfp4-gptq-v17   Host path to checkpoint

services:
  qwen38-27b-v17:
    image: lribeiro/qwen38-quant:ii-r15-p1 # ii with mxfp4 patches and other minor corrections
    container_name: qwen38-27b-v17
    ipc: host
    ports:
      - "${PORT:-8000}:${PORT:-8000}"
    volumes:
      # Mount the quantized model checkpoint (read-only)
      - ${MODEL_DIR:-./models/Qwen3.8-27B-nvfp4-gptq-v17}:/models/Qwen3.8-27B-nvfp4-gptq-v17:ro
      # Persistent JIT / compile caches (avoids recompilation on restart)
      - vllm-cache:/cache
      # HuggingFace hub cache (tokenizer / config downloads)
      - hf-cache:/data/hf_cache
    environment:
      # ── GPU / runtime ──
      - CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES:-0}
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
      - SAFETENSORS_FAST_GPU=1
      - OMP_NUM_THREADS=16
      # ── vLLM engine ──
      - VLLM_USE_AOT_COMPILE=1
      - VLLM_USE_FLASHINFER_SAMPLER=1
      - VLLM_USE_V2_MODEL_RUNNER=1
      - VLLM_ALLOW_INSECURE_SERIALIZATION=1
      # SM120: disable broken FP4/FP8 kernels — use FlashInfer / b12x /  CUTLASS paths
      - VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel,MarlinFP8ScaledMMLinearKernel
      # ── Cache directories ──
      - HF_HOME=/data/hf_cache
      - VLLM_CACHE_DIR=/cache/vllm
      - TORCHINDUCTOR_CACHE_DIR=/cache/torchinductor
      - TRITON_CACHE_DIR=/cache/triton
      - FLASHINFER_WORKSPACE_BASE=/cache/flashinfer
    entrypoint: ["python", "-m", "vllm.entrypoints.cli.main"]
    command:
      - serve
      - /models/Qwen3.8-27B-nvfp4-gptq-v17
      - --served-model-name=${SERVED_MODEL_NAME:-qwen38-27b}
      - --host=0.0.0.0
      - --port=${PORT:-8000}
      - --trust-remote-code
      - --quantization=compressed-tensors
      - --tensor-parallel-size=${TENSOR_PARALLEL_SIZE:-1}
      - --kv-cache-dtype=${KV_CACHE_DTYPE:-fp8}
      - --block-size=128
      - --load-format=fastsafetensors
      - --gpu-memory-utilization=${GPU_MEMORY_UTILIZATION:-0.94}
      - --max-model-len=${MAX_MODEL_LEN:-262144}
      - --max-num-seqs=${MAX_NUM_SEQS:-512}
      - --max-num-batched-tokens=${MAX_NUM_BATCHED_TOKENS:-8192}
      - --generation-config=vllm
      - --reasoning-parser=qwen3
      - --attention-backend=flashinfer
      - --linear-backend=auto
      - --max-cudagraph-capture-size=16
      - --async-scheduling
      - --enable-chunked-prefill
      - --enable-prefix-caching
      - --enable-flashinfer-autotune
      - >-
        --compilation-config={"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]

volumes:
  vllm-cache:
  hf-cache:

Configurable variables (set in .env or shell):

Table with columns: Variable, Default, Description
Variable	Default	Description
`PORT`	`8000`	API server port (OpenAI-compatible)
`SERVED_MODEL_NAME`	`qwen38-27b`	Model name returned by `/v1/models`
`MAX_MODEL_LEN`	`262144`	Maximum context length in tokens

Evaluation

Distribution Fidelity (KLD)

Table with columns: Metric, Value
Metric	Value
Token mean KLD	0.012310
Token median KLD	0.002351
p95 KLD	0.047520
p99 KLD	0.191077
p99.9 KLD	0.694279
Max KLD	3.961384
Mean JSD (bits)	0.004245
Top-1 agreement	96.17%
Validation tier

KLD by Domain

Table with columns: Stratum, Contexts, Mean KLD
Stratum	Contexts	Mean KLD
scientific	39	0.00409
multilingual	7	0.00439
encyclopedic	13	0.00866
code	36	0.01276
literary	41	0.02224

Throughput

Single NVIDIA RTX PRO 6000 Blackwell (SM120, 96 GB), vLLM 0.26.1rc0, FlashInfer backend. Duration-based sustained decode, 30s cells, temperature 0.0, ignore_eos=true.

Table with columns: Metric, Value
Metric	Value
Prefill (2k context)	7,306 tok/s
TTFT (2k)	0.279 s
Decode C1 (0 ctx)	49.8 tok/s
Decode C1 (2k ctx)	49.4 tok/s
Decode C4 (0 ctx)	180.6 tok/s
Decode C4 (2k ctx)	176.1 tok/s

Composite Score

$score = KLD \times 106 prefill 2 k \times decode c 1 =$

Context Within Experiment Series

Table with columns: Experiment, Change vs v17, KLD, vs v17, Score
Experiment	Change vs v17	KLD	vs v17	Score
v17 (baseline)	—	0.01231	—	29.55
v23	damp=0.001	0.01241	+1%	29.03
v24	damp=0.005, block256	0.01253	+2%	28.52

GPTQ dampening is irrelevant for FP8. Tested 0.001–0.1 — all produce 0.0124 ± noise. The GPTQ Hessian is well-conditioned for FP8 regardless of dampening_frac.

Calibration sample count doesn't matter. v16 (512 samples) vs v17 (181 samples) — KLD 0.01245 vs 0.01231, within noise. The Hessian is well-estimated at 181 samples.

Pareto Position

Table with columns: Objective, Best Model, KLD, Prefill, Decode, Score
Objective	Best Model	KLD	Prefill	Decode	Score
Best composite score	v31	0.01014	6,385	49.9	31.45
Best all-FP8 (uniform)	v17	0.01231	7,306	49.8	29.55
Lowest KLD (our quants)

Provenance & Integrity

Table with columns: Artifact, SHA-256
Artifact	SHA-256
Model index	`aa62fb91085564f542299f80c1447e044fca41fbb89350d2979adc0e2804e097`
Config	`dc8f1def22f7c0f5ed182444c8431d16dbdc9aab398c894918c864e96c78ad74`
Suite tokens	`3f9d17f1b55f64872ad3ac19c8711654e09ba70b7ca14b0851525088fe735691`
Capture manifest	`fb2317dd353c01b4544427ded1c2966b4276e370ae2d1b492ac052858ce9d997`
Shared LM head	`25a30fd5f826da0abc4efc4cc71def9f02bcb8085f7175eee284d221dee4cfff`

Hardware & Runtime

Table with columns: Field, Value
Field	Value
GPU	4× NVIDIA RTX PRO 6000 Blackwell Max-Q (SM120, 96 GiB GDDR7 each)
Driver	595.58.03
Benchmark GPU count	1
VRAM used	86,606 / 97,887 MB (88.5%)
Temp (avg / max)	53.0 °C / 59.0 °C

Reproduction

Step 1: Original v17 quantization (already done)

bash
# Inside Docker (vLLM infernal-invocation image, FlashInfer, SM120)
CUDA_VISIBLE_DEVICES=0 python /data/scripts/quantize_nvfp4_gptq_v17.py \
  --model /data/models/Qwen3.8-27B-bf16 \
  --output /data/models/Qwen3.8-27B-nvfp4-gptq-v17 \
  --calib-samples 181 \
  --calib-max-len 2048

Calibration tokens from $SUITE_DIR/tokens (default /data/suite-v3/tokens). Script: scripts/quantize_nvfp4_gptq_v17.py. Log: quant_v17.log.

Step 2: Merge visual tower + MTP from base (this checkpoint)

bash
# Merge v17 quantized LM shards with BF16 visual+mtp from base model
python /workspace/scripts/merge_v17_vision.py \
  --v17 /models/Qwen3.8-27B-nvfp4-gptq-v17 \
  --base /models/Qwen3.8-27B-bf16 \
  --output /data/models/Qwen3.8-27B-nvfp4-gptq-v17-vision

Note: The v17 script's default --calib-samples is 1024, but the suite contains only 181 contexts. llm-compressor warns
markdown
Requested 1024 samples but the provided dataset only has 181 samples
and uses all 181. The effective calibration set is 181 samples, matching v6/v16/v31. Pass --calib-samples 181 to suppress the warning.

Limitations

KLD far from malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated
Superseded by v31. v31 (FP8 W8A8 MLP + W8A16 weight-only attention) achieves 18% lower KLD (0.01014 vs 0.01231) at similar decode speed. Use v17 when uniform FP8 is required or when the faster prefill (7,306 vs 6,385 tok/s) matters more than the KLD gap.
Attention activation quantization is the second-largest KLD contributor. FP8 W8A8 quantizes attention activations; v31 eliminates this by keeping BF16 activations for attention (W8A16 weight-only). KLD attribution: MLP ~0.009, linear_attn ~0.0025, self_attn ~0.0006.
Blackwell-optimized. FP8 E4M3 MMA is supported on SM120 (Blackwell), SM89/SM90 (Ada/Hopper), but the benchmark and KLD capture were run on SM120 only. Validate on your hardware.
Body-only KLD. Fidelity measured against a shared BF16 LM head; end-to-end generation quality may differ slightly.
Literary domain degradation. 5.4× higher KLD on literary text vs scientific — creative/prose generation is disproportionately affected, though less severely than in 4-bit configs.
Quantization is lossy. Validate on your own workload before production use.

Qwen3.8-27B-nvfp4-v17

README

Model Details

Quantization Configuration

Precision Assignment

Modules Kept in BF16

Calibration

Usage

Docker Compose

Evaluation

Distribution Fidelity (KLD)

KLD by Domain

Throughput

Composite Score

Context Within Experiment Series

Pareto Position

Provenance & Integrity

Hardware & Runtime

Reproduction

Step 1: Original v17 quantization (already done)

Step 2: Merge visual tower + MTP from base (this checkpoint)

Limitations

Acknowledgements

Explore FriendliAI today

README

Model Details

Quantization Configuration

Precision Assignment

Modules Kept in BF16

Calibration

Usage

Docker Compose

Evaluation

Distribution Fidelity (KLD)

KLD by Domain

Throughput

Composite Score

Context Within Experiment Series

Pareto Position

Provenance & Integrity

Hardware & Runtime

Reproduction

Step 1: Original v17 quantization (already done)

Step 2: Merge visual tower + MTP from base (this checkpoint)

Limitations

Acknowledgements