WaveCut

Qwen3-VL-8B-Instruct-SDNQ-int8-static

README

License: apache-2.0

Recipe

SDNQ 0.2.5 static quantization, weights_dtype=int8, group_size=0 (auto), dequantize_fp32=True.
Kept in bf16: embeddings, lm_head, multi_modal_projector, vision patch_embed and every module below SDNQ's size floor.
Quantized layers: {"int8": 368}; quantized params: {"int8": 7517270016}; saved size 9.37 GiB (bf16 source 16.33 GiB).
Save/load round trip verified through sdnq.loader.load_sdnq_model (all parameters bit-identical); encoder last_hidden_state vs bf16 on a mixed RU/EN/ZH prompt: {"cos_mean": 0.9716747403144836, "cos_min": 0.7031210064888, "max_abs": 30.71875, "rel_l2": 0.24023021757602692}.
Runtime used for all numbers below: torch 2.10.0+cu130, transformers 4.57.1, diffusers 0.39.0.dev0, sdnq 0.2.5, NVIDIA GeForce RTX 4090 (RunPod, disposable pod), Python 3.12.3.

Boogu Image 0.1 Turbo A/B (same seed, bf16 encoder vs this encoder)

Pipeline BooguImageTurboPipeline from WaveCut/Boogu-Image-0.1-Turbo-SDNQ-uint4-static @ 422ac37ad305169703fd8f847ef76418cab8363f (SDNQ uint4 transformer), 1024x1024, 4 DMD steps, text_guidance_scale=1.0, image_guidance_scale=1.0, empty_instruction_guidance_scale=0.0, dmd_conditioning_sigma=0.001, max_sequence_length=1280, sequential CPU offload, SDNQ quantized matmul on for the transformer. 12 prompts (EN/RU/ZH, typography, diagrams, long prompt); image metrics are against the bf16-encoder image of the same prompt and seed, embedding metrics compare the encoder output that is fed to the transformer. Encoder rows ending in -wo run weight-only (quantized weights dequantized per layer on the GPU, use_quantized_matmul=False); rows without the suffix also quantize the activations to int8 for the matmul, which is measurably worse for this encoder (Qwen3-VL activation outliers) and is not the recommended mode.

Table with columns: Encoder, Gen mean s, Encode mean s, Peak VRAM MiB, Host RSS after load GiB, RSS anon GiB, PSNR, SSIM, LPIPS, Embed cos, Embed rel-L2
Encoder	Gen mean s	Encode mean s	Peak VRAM MiB	Host RSS after load GiB	RSS anon GiB	PSNR	SSIM	LPIPS	Embed cos	Embed rel-L2
bf16 (reference)	20.01	1.27	2446	1.33	0.83	-	-	-

Host RSS is the whole pipeline process (transformer + VAE + encoder) after load with sequential offload, measured from /proc/self/status.

Per prompt (int8-wo)

Table with columns: Prompt, PSNR, SSIM, LPIPS, Embed cos, Embed rel-L2
Prompt	PSNR	SSIM	LPIPS	Embed cos	Embed rel-L2
00_vase	34.89	0.9766	0.0138	0.99899	0.03695
01_portrait	18.32	0.6910	0.2393	0.99904	0.03652
02_abstract	19.80	0.6543

Pairwise images (bf16 | quantized) are in comparison/.

Usage

python
import torch
import sdnq  # registers the SDNQ quantizer with transformers
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

repo = "WaveCut/Qwen3-VL-8B-Instruct-SDNQ-int8-static"
processor = AutoProcessor.from_pretrained(repo)
model = Qwen3VLForConditionalGeneration.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda").eval()

Loader used by the aifarm draw service (weights stay on CPU, sequential offload moves layers per forward):

python
from sdnq.loader import load_sdnq_model
from huggingface_hub import snapshot_download

# model_cls=None: sdnq 0.2.5 crashes on an explicit transformers class (fixed in 0.2.6);
# use_quantized_matmul=False: weight-only, keeps the encoder output at cos ~0.999 vs bf16.
mllm = load_sdnq_model(snapshot_download(repo), model_cls=None,
                       dtype=torch.bfloat16, device="cpu", use_quantized_matmul=False)

Files

model-*.safetensors, config.json (with quantization_config), quantization_config.json - SDNQ checkpoint.
quantization_summary.json - recipe, layer counts, storage, save/load checks.
benchmark/summary.json, benchmark/*.metrics.json - raw A/B metrics; benchmark/prompts.json - the prompt set.
comparison/ - pairwise images and the contact sheet.
SHA256SUMS - checksums of the checkpoint files.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

WaveCut

Model Tree

Base

Qwen/Qwen3-VL-8B-Instruct

Quantized

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer