nameistoken

Qwen3.6-27B-Quark-W8A8-INT8

README

License: apache-2.0

Model Details

Table

Base Model	`Qwen/Qwen3.6-27B`
Architecture	`Qwen3_5ForConditionalGeneration` (hybrid attention + ViT)
Parameters	27B language tower (quantized) + 27-layer ViT (BF16, unquantized)
Layers	64 hybrid (16 full_attention + 48 linear_attention GatedDeltaNet) + 1 MTP head
Quantization	W8A8 INT8 (per-channel weight + per-token dynamic activation)
Quantizer	AMD Quark `0.11.1` (`pack_method='reorder'`, vLLM-native key naming)
Model Size	~29 GB (single safetensors)
Original Size	~52 GB (BF16)
Compression	~1.8x size reduction

Quantization Scheme

Table with columns: Component, dtype, Granularity, Mode
Component	dtype	Granularity	Mode
Linear weight (text decoder)	INT8	per-channel (`ch_axis=0`)	symmetric, static
Linear activation	INT8	per-token (`ch_axis=1`)	symmetric, dynamic
`lm_head`	BF16	-	unquantized

Accuracy

GSM8K full 1319-question test split (vLLM, temperature=0, concurrency=16, max_tokens=1024, chat_template_kwargs.enable_thinking=false):

Table with columns: Model, Accuracy, Correct
Model	Accuracy	Correct
`Qwen/Qwen3.6-27B` (BF16 baseline)	96.74%	1276 / 1319
This model (Quark W8A8 INT8)	96.74%	1276 / 1319

Net accuracy delta vs BF16: 0.00 pp.

Although the totals match exactly, the two models diverge on individual questions: only 38 / 1319 generations are token-identical, and the correct-set Jaccard is 0.9891 (1269 common correct, BF16 wins 7 unique, INT8 wins 7 unique — they cancel out). This is the typical W8A8 INT8 pattern: small per-token numerical drift causes reasoning paths to fork, but the accuracy averages out with no systematic degradation.

Eval setup: vLLM /v1/chat/completions, temperature=0, concurrency=16, max_tokens=1024, chat_template_kwargs.enable_thinking=false, single MI355X GPU (TP=1) for INT8 / TP=8 for BF16.

How to Use

With vLLM (Recommended)

bash
vllm serve nameistoken/Qwen3.6-27B-Quark-W8A8-INT8 \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.9 \
    --trust-remote-code

Chat completion call:

bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nameistoken/Qwen3.6-27B-Quark-W8A8-INT8",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 256, "temperature": 0.7,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Hardware Requirements

~32 GB VRAM minimum (e.g., AMD MI300X / MI355X, NVIDIA A100-40G or larger).

Quantization Details

This model was quantized using AMD Quark's per-token per-channel INT8 scheme:

Weight: INT8 per-channel symmetric static (PerChannelMinMaxObserver, ch_axis=0).
Activation: INT8 per-token symmetric dynamic (ch_axis=1).
Excluded layers: lm_head, *embed_tokens*, *visual*, mtp*.
Export: pack_method='reorder', weight_format='real_quantized', custom_mode='quark'.
Key-name post-process: *.weight_quantizer.scale → *.weight_scale, drop (symmetric). Required for vLLM path with transformers 5.x.

License

Apache License 2.0 (inherited from Qwen/Qwen3.6-27B). See LICENSE and NOTICE.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

nameistoken

Model Tree

Base

Qwen/Qwen3.6-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Table

Base Model	`Qwen/Qwen3.6-27B`
Architecture	`Qwen3_5ForConditionalGeneration` (hybrid attention + ViT)
Parameters	27B language tower (quantized) + 27-layer ViT (BF16, unquantized)
Layers	64 hybrid (16 full_attention + 48 linear_attention GatedDeltaNet) + 1 MTP head
Quantization	W8A8 INT8 (per-channel weight + per-token dynamic activation)
Quantizer	AMD Quark `0.11.1` (`pack_method='reorder'`, vLLM-native key naming)
Model Size	~29 GB (single safetensors)
Original Size	~52 GB (BF16)
Compression	~1.8x size reduction

Quantization Scheme

Table with columns: Component, dtype, Granularity, Mode
Component	dtype	Granularity	Mode
Linear weight (text decoder)	INT8	per-channel (`ch_axis=0`)	symmetric, static
Linear activation	INT8	per-token (`ch_axis=1`)	symmetric, dynamic
`lm_head`	BF16	-	unquantized

Accuracy

GSM8K full 1319-question test split (vLLM, temperature=0, concurrency=16, max_tokens=1024, chat_template_kwargs.enable_thinking=false):

Table with columns: Model, Accuracy, Correct
Model	Accuracy	Correct
`Qwen/Qwen3.6-27B` (BF16 baseline)	96.74%	1276 / 1319
This model (Quark W8A8 INT8)	96.74%	1276 / 1319

Net accuracy delta vs BF16: 0.00 pp.

Eval setup: vLLM /v1/chat/completions, temperature=0, concurrency=16, max_tokens=1024, chat_template_kwargs.enable_thinking=false, single MI355X GPU (TP=1) for INT8 / TP=8 for BF16.

How to Use

With vLLM (Recommended)

bash
vllm serve nameistoken/Qwen3.6-27B-Quark-W8A8-INT8 \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.9 \
    --trust-remote-code

Chat completion call:

bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nameistoken/Qwen3.6-27B-Quark-W8A8-INT8",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 256, "temperature": 0.7,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Hardware Requirements

~32 GB VRAM minimum (e.g., AMD MI300X / MI355X, NVIDIA A100-40G or larger).

Quantization Details

This model was quantized using AMD Quark's per-token per-channel INT8 scheme:

Weight: INT8 per-channel symmetric static (PerChannelMinMaxObserver, ch_axis=0).
Activation: INT8 per-token symmetric dynamic (ch_axis=1).
Excluded layers: lm_head, *embed_tokens*, *visual*, mtp*.
Export: pack_method='reorder', weight_format='real_quantized', custom_mode='quark'.
Key-name post-process: *.weight_quantizer.scale → *.weight_scale, drop (symmetric). Required for vLLM path with transformers 5.x.

License

Apache License 2.0 (inherited from Qwen/Qwen3.6-27B). See LICENSE and NOTICE.