88plug

Qwen3-Omni-30B-A3B-W8A16

README

License: apache-2.0

At a Glance

Table with columns: Property, Value
Property	Value
Base model	`Qwen/Qwen3-30B-A3B`
Release tier	Provisional (datafree RTN — re-quant scheduled)
Quant method	datafree RTN W8A16 (weight-only INT8)
FLAC status	Not measured (T+7d milestone)
Architecture	Sparse MoE + Whisper audio + ViT vision + speech decoder
Quant format	compressed-tensors (native vLLM)
Quantized	48 attention layers (q/k/v/o_proj) — 192 tensors total
Kept BF16	audio_tower, visual, talker, code2wav
Disk size	~33 GB
Min GPU	1× A6000 48GB or A100 40GB

Why only attention projections?

Qwen3-Omni's MoE experts are implemented via @use_experts_implementation as stacked nn.Parameter tensors — not nn.Linear submodules. They cannot be reached by targets=["Linear"] in any current quantization framework. Only the 192 attention projections (48 layers × q/k/v/o_proj) are quantizable via standard W8A16 RTN.

Memory Requirements

Table with columns: Configuration, BF16, W8A16
Configuration	BF16	W8A16
Weights	~66 GB	~33 GB
Min GPU	2× A100 40GB	1× A100 40GB / A6000 48GB

Quick Start

Tested with vLLM v0.21.0 (vllm/vllm-openai:v0.21.0-cu129-ubuntu2404). Weights are in compressed-tensors format — vLLM detects and loads quantization automatically. No --quantization flag needed.

vLLM — text output only

bash

docker run --gpus device=0 -p 8080:8080 \  vllm/vllm-openai:v0.21.0-cu129-ubuntu2404 vllm serve \  88plug/Qwen3-Omni-30B-A3B-W8A16 \  --kv-cache-dtype fp8 \  --max-model-len 32768 \  --gpu-memory-utilization 0.92

Weights are in compressed-tensors format — no --quantization flag needed. Requires vLLM ≥ v0.21.0. Mainline vLLM returns text only. Audio input works; speech output does not.

vLLM-Omni — full audio output

vLLM-Omni v0.20.0 enables real-time speech output. Required if you need the model to speak.

bash

docker run --gpus device=0 -p 8080:8080 \  vllm/vllm-omni:v0.20.0-cu129 vllm serve \  88plug/Qwen3-Omni-30B-A3B-W8A16 \  --kv-cache-dtype fp8 \  --max-model-len 32768 \  --gpu-memory-utilization 0.92

Recommended Sampling Parameters

Table with columns: Mode, Temperature, Top-P, Top-K, Min-P, Use When
Mode	Temperature	Top-P	Top-K	Min-P	Use When
Thinking (default)	0.6	0.95	20	0.0	Reasoning, math, code
Non-thinking	0.7	0.8	20	0.0	Chat, creative, fast response

Enable/disable thinking via chat_template_kwargs={"enable_thinking": True/False}. Default is thinking-enabled.

llama.cpp — audio/vision in, text out (no speech output in mainline)

Speech output is not yet in mainline llama.cpp (issue #21956). For full speech out, use the tc-mb/llama.cpp-omni fork. For audio/vision input with text output, mainline b9297 works.

bash

# Convert from BF16 base (audio/vision inputs supported via libmtmd)python convert_hf_to_gguf.py Qwen/Qwen3-30B-A3B \  --outfile Qwen3-Omni-30B-BF16.gguf llama-quantize Qwen3-Omni-30B-BF16.gguf Qwen3-Omni-30B-Q8_0.gguf Q8_0 llama-server \  --model Qwen3-Omni-30B-Q8_0.gguf \  --n-gpu-layers 999 \  --ctx-size 32768 \  --port 8081

Benchmarks

Table with columns: Metric, Status
Metric	Status
Throughput (tok/s)	In progress — T+7d milestone
MMLU delta vs BF16	In progress — T+7d milestone
RULER@128k	In progress — T+30d milestone

No fabricated numbers. Results will be published to this card when measured.

What's Quantized, What's Not

Table with columns: Component, Precision, Reason
Component	Precision	Reason
Attention q/k/v/o_proj (all 48 layers)	W8A16 INT8	Quantized
MoE experts (routed + shared)	BF16	Not reachable as nn.Linear
`thinker.audio_tower.*`	BF16	Whisper encoder — excluded
`thinker.visual.*`	BF16	ViT — excluded
`talker.*`

Quality Targets

Table with columns: Metric, Target
Metric	Target
KL divergence vs BF16	< 0.005
MMLU recovery	≥ 99.7%
RULER@128k recovery	≥ 99%

vs. Other Qwen3-Omni-30B Quants

This is the first compressed-tensors W8A16 checkpoint for Qwen3-Omni-30B-A3B. It targets the gap between the full BF16 model and hardware-restricted FP8, enabling single-GPU deployment on 48GB cards.

Table with columns: Quant, Method, Size, GPU Compatibility, Notes
Quant	Method	Size	GPU Compatibility	Notes
88plug W8A16 (this)	compressed-tensors RTN W8A16	~33 GB	Any Ampere+	First W8A16; native vLLM; attention-only quant
unsloth Qwen3-Omni-30B GGUF	llama.cpp GGUF Q4_K_M	~19 GB	CPU / any GPU	Text+audio in; no speech output in mainline
unsloth Qwen3-Omni-30B GGUF	llama.cpp GGUF Q8_0	~34 GB	Any GPU ≥8 GB

Limitations

Attention-only quantization: Qwen3-Omni's MoE expert weights are stacked nn.Parameter tensors, not nn.Linear submodules — unreachable by any current quantization framework. Only the 192 attention projections are quantized. Expert weights remain BF16.
No speech output in mainline vLLM: talker and code2wav TTS components stay BF16 and are not supported by mainline vLLM. Use vLLM-Omni v0.20.0 for speech output.
Multimodal encoders excluded: audio_tower (Whisper) and visual (ViT) stay BF16 — RTN INT8 not applied.
Benchmark results pending: Throughput and quality benchmarks (MMLU-Pro, GPQA) will be added post-publication.

Citation

bibtex

@misc{qwen3technicalreport,  title  = {Qwen3 Technical Report},  author = {Qwen Team},  year   = {2025},  url    = {https://huggingface.co/Qwen/Qwen3-30B-A3B}}

About

88plug AI Lab ships compressed-tensors quantizations for native vLLM v0.21.0+ deployment.

This release: Provisional tier — datafree RTN (weight-only rounding, no calibration corpus). A gold AutoRound re-quant is scheduled; 88plug architecture forbids new provisional W4A16 uploads.

Browse all releases → huggingface.co/88plug

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

88plug

Model Tree

Base

Qwen/Qwen3-Omni-30B-A3B-Instruct

Quantized

this model

Input Modalities

TextAudioImageVideo

Output Modalities

TextAudio

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

At a Glance

Table with columns: Property, Value
Property	Value
Base model	`Qwen/Qwen3-30B-A3B`
Release tier	Provisional (datafree RTN — re-quant scheduled)
Quant method	datafree RTN W8A16 (weight-only INT8)
FLAC status	Not measured (T+7d milestone)
Architecture	Sparse MoE + Whisper audio + ViT vision + speech decoder
Quant format	compressed-tensors (native vLLM)
Quantized	48 attention layers (q/k/v/o_proj) — 192 tensors total
Kept BF16	audio_tower, visual, talker, code2wav
Disk size	~33 GB
Min GPU	1× A6000 48GB or A100 40GB

Why only attention projections?

Memory Requirements

Table with columns: Configuration, BF16, W8A16
Configuration	BF16	W8A16
Weights	~66 GB	~33 GB
Min GPU	2× A100 40GB	1× A100 40GB / A6000 48GB

Quick Start

vLLM — text output only

bash

docker run --gpus device=0 -p 8080:8080 \  vllm/vllm-openai:v0.21.0-cu129-ubuntu2404 vllm serve \  88plug/Qwen3-Omni-30B-A3B-W8A16 \  --kv-cache-dtype fp8 \  --max-model-len 32768 \  --gpu-memory-utilization 0.92

Weights are in compressed-tensors format — no --quantization flag needed. Requires vLLM ≥ v0.21.0. Mainline vLLM returns text only. Audio input works; speech output does not.

vLLM-Omni — full audio output

vLLM-Omni v0.20.0 enables real-time speech output. Required if you need the model to speak.

bash

docker run --gpus device=0 -p 8080:8080 \  vllm/vllm-omni:v0.20.0-cu129 vllm serve \  88plug/Qwen3-Omni-30B-A3B-W8A16 \  --kv-cache-dtype fp8 \  --max-model-len 32768 \  --gpu-memory-utilization 0.92

Recommended Sampling Parameters

Table with columns: Mode, Temperature, Top-P, Top-K, Min-P, Use When
Mode	Temperature	Top-P	Top-K	Min-P	Use When
Thinking (default)	0.6	0.95	20	0.0	Reasoning, math, code
Non-thinking	0.7	0.8	20	0.0	Chat, creative, fast response

Enable/disable thinking via chat_template_kwargs={"enable_thinking": True/False}. Default is thinking-enabled.

llama.cpp — audio/vision in, text out (no speech output in mainline)

Speech output is not yet in mainline llama.cpp (issue #21956). For full speech out, use the tc-mb/llama.cpp-omni fork. For audio/vision input with text output, mainline b9297 works.

bash

# Convert from BF16 base (audio/vision inputs supported via libmtmd)python convert_hf_to_gguf.py Qwen/Qwen3-30B-A3B \  --outfile Qwen3-Omni-30B-BF16.gguf llama-quantize Qwen3-Omni-30B-BF16.gguf Qwen3-Omni-30B-Q8_0.gguf Q8_0 llama-server \  --model Qwen3-Omni-30B-Q8_0.gguf \  --n-gpu-layers 999 \  --ctx-size 32768 \  --port 8081

Benchmarks

Table with columns: Metric, Status
Metric	Status
Throughput (tok/s)	In progress — T+7d milestone
MMLU delta vs BF16	In progress — T+7d milestone
RULER@128k	In progress — T+30d milestone

No fabricated numbers. Results will be published to this card when measured.

What's Quantized, What's Not

Table with columns: Component, Precision, Reason
Component	Precision	Reason
Attention q/k/v/o_proj (all 48 layers)	W8A16 INT8	Quantized
MoE experts (routed + shared)	BF16	Not reachable as nn.Linear
`thinker.audio_tower.*`	BF16	Whisper encoder — excluded
`thinker.visual.*`	BF16	ViT — excluded
`talker.*`

Quality Targets

Table with columns: Metric, Target
Metric	Target
KL divergence vs BF16	< 0.005
MMLU recovery	≥ 99.7%
RULER@128k recovery	≥ 99%

vs. Other Qwen3-Omni-30B Quants

This is the first compressed-tensors W8A16 checkpoint for Qwen3-Omni-30B-A3B. It targets the gap between the full BF16 model and hardware-restricted FP8, enabling single-GPU deployment on 48GB cards.

Table with columns: Quant, Method, Size, GPU Compatibility, Notes
Quant	Method	Size	GPU Compatibility	Notes
88plug W8A16 (this)	compressed-tensors RTN W8A16	~33 GB	Any Ampere+	First W8A16; native vLLM; attention-only quant
unsloth Qwen3-Omni-30B GGUF	llama.cpp GGUF Q4_K_M	~19 GB	CPU / any GPU	Text+audio in; no speech output in mainline
unsloth Qwen3-Omni-30B GGUF	llama.cpp GGUF Q8_0	~34 GB	Any GPU ≥8 GB

Limitations

Attention-only quantization: Qwen3-Omni's MoE expert weights are stacked nn.Parameter tensors, not nn.Linear submodules — unreachable by any current quantization framework. Only the 192 attention projections are quantized. Expert weights remain BF16.
No speech output in mainline vLLM: talker and code2wav TTS components stay BF16 and are not supported by mainline vLLM. Use vLLM-Omni v0.20.0 for speech output.
Multimodal encoders excluded: audio_tower (Whisper) and visual (ViT) stay BF16 — RTN INT8 not applied.
Benchmark results pending: Throughput and quality benchmarks (MMLU-Pro, GPQA) will be added post-publication.

Citation

bibtex

@misc{qwen3technicalreport,  title  = {Qwen3 Technical Report},  author = {Qwen Team},  year   = {2025},  url    = {https://huggingface.co/Qwen/Qwen3-30B-A3B}}

About

88plug AI Lab ships compressed-tensors quantizations for native vLLM v0.21.0+ deployment.

This release: Provisional tier — datafree RTN (weight-only rounding, no calibration corpus). A gold AutoRound re-quant is scheduled; 88plug architecture forbids new provisional W4A16 uploads.

Browse all releases → huggingface.co/88plug