hongweimeng

Qwen3.6-35B-A3B-RTN-UINT4-W4A16

README

License: apache-2.0

Quantization

Table with columns: Item, Value
Item	Value
Scheme	`uint4_wo_128` asymmetric, group size 128
Method	RTN (round-to-nearest, no calibration)
Pack	Quark `real_quantized`, `pack_method=reorder`
Quantized	MoE experts + `self_attn` + `linear_attn`
FP16 kept	`mlp.gate`, shared expert, `lm_head`, vision, MTP

Evaluation (Wikitext-2 test)

Table with columns: Metric, Value
Metric	Value
Perplexity	6.926
Tokens scored	296,815
Chunks (seq_len=2048)	145
Eval time	~252 s

See eval/ppl_wikitext.json for full results.

Evaluation (Open LLM Leaderboard v2, lm_eval)

Evaluated with EleutherAI/lm-evaluation-harness --tasks leaderboard via vLLM on 2026-07-22. All six leaderboard-v2 tasks were measured on this model. Run config: tensor_parallel_size=4, gpu_memory_utilization=0.90, max_model_len=8192, VLLM_USE_TRITON_AWQ=1.

Table with columns: Task, Metric, BF16 (base), UINT4 (this), INT4
Task	Metric	BF16 (base)	UINT4 (this)	INT4
IFEval	prompt_level_strict	0.3235	0.3383	0.3438
IFEval	inst_level_strict	0.4580	0.4664	0.4676
MATH-hard	exact_match	0.5287	0.4856	0.5083

Per-task raw metrics are shown (the six tasks use different metrics, so no single average is reported). Full result JSON and per-sample JSONL files are in eval/leaderboard/.

Evaluation (GSM8K, 5-shot)

Evaluated alongside the BF16 base model and the INT4 variant under identical conditions. The prompt suffix <think>\n\n</think>\n\n is prepended to each answer field to suppress the model's thinking preamble so all three runs are comparable (see custom task yaml below).

Table with columns: Model, flexible-extract, strict-match
Model	flexible-extract	strict-match
Qwen3.6-35B-A3B BF16 (base)	0.9158 ± 0.0076	0.9090 ± 0.0079
Qwen3.6-35B-A3B RTN-INT4-W4A16	0.9045 ± 0.0081	0.8961 ± 0.0084
Qwen3.6-35B-A3B RTN-UINT4-W4A16 (this model)	0.9022 ± 0.0082	0.8976 ± 0.0083

Eval command:

bash

# 1. Create a custom task yaml that suppresses thinking (save as gsm8k_nothink/gsm8k_nothink.yaml)cat << 'EOF' > gsm8k_nothink/gsm8k_nothink.yamltask: gsm8k_nothinkdataset_path: openai/gsm8kdataset_name: mainoutput_type: generate_untiltraining_split: trainfewshot_split: traintest_split: testdoc_to_text: "Question: {{question}}\nAnswer: <think>\n\n</think>\n\n"doc_to_target: "{{answer}}"metric_list:  - metric: exact_match    aggregation: mean    higher_is_better: true    ignore_case: true    ignore_punctuation: false    regexes_to_ignore:      - ","      - "\\$"      - "(?s).*#### "      - "\\.$"generation_kwargs:  until:    - "Question:"    - "</s>"    - "<|im_end|>"  do_sample: false  temperature: 0.0repeats: 1num_fewshot: 5filter_list:  - name: "strict-match"    filter:      - function: "regex"        regex_pattern: "#### (\\-?[0-9\\.\\,]+)"      - function: "take_first"  - name: "flexible-extract"    filter:      - function: "regex"        group_select: -1        regex_pattern: "(-?[$0-9.,]{2,})|(-?[0-9]+)"      - function: "take_first"metadata:  version: 3.0EOF # 2. Run evaluationCUDA_VISIBLE_DEVICES=0 \VLLM_USE_TRITON_AWQ=1 \lm_eval run \  --model vllm \  --model_args "pretrained=hongweimeng/Qwen3.6-35B-A3B-RTN-UINT4-W4A16,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.95,enforce_eager=True" \  --tasks gsm8k_nothink \  --include_path . \  --num_fewshot 5 \  --batch_size auto

VLLM_USE_TRITON_AWQ=1 is required — the default CUDA AWQ kernel enforces OC % group_size == 0, which some MoE expert layers violate at small batch sizes. The Triton kernel has no such restriction.

vLLM inference

Requires vLLM with Quark W4A16 UINT4 support. Set VLLM_USE_TRITON_AWQ=1 to use the Triton AWQ kernel, which handles all layer shapes correctly (see GSM8K note above).

python

import osos.environ["VLLM_USE_TRITON_AWQ"] = "1" from vllm import LLM, SamplingParams llm = LLM(    model="hongweimeng/Qwen3.6-35B-A3B-RTN-UINT4-W4A16",    quantization="quark",    dtype="auto",    gpu_memory_utilization=0.95,    enforce_eager=True,)print(llm.generate(["Hello"], SamplingParams(max_tokens=32))[0].outputs[0].text)

Reproduce

This repo includes scripts/ used to quantize and evaluate:

bash

source scripts/env.shbash scripts/01_quantize_rtn_uint4.sh   # needs base model + quark + GPUbash scripts/02_eval_ppl.sh

Base model: Qwen/Qwen3.6-35B-A3B (~67 GB FP16).

Files

model.safetensors — quantized weights (~21 GB)
config.json — includes quantization_config with layer excludes
scripts/ — quantization and PPL evaluation scripts
eval/ppl_wikitext.json — Wikitext-2 PPL results
eval/leaderboard/ — Open LLM Leaderboard results (JSON + per-sample JSONL)

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

hongweimeng

Model Tree

Base

Qwen/Qwen3.6-35B-A3B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Quantization

Table with columns: Item, Value
Item	Value
Scheme	`uint4_wo_128` asymmetric, group size 128
Method	RTN (round-to-nearest, no calibration)
Pack	Quark `real_quantized`, `pack_method=reorder`
Quantized	MoE experts + `self_attn` + `linear_attn`
FP16 kept	`mlp.gate`, shared expert, `lm_head`, vision, MTP

Evaluation (Wikitext-2 test)

Table with columns: Metric, Value
Metric	Value
Perplexity	6.926
Tokens scored	296,815
Chunks (seq_len=2048)	145
Eval time	~252 s

See eval/ppl_wikitext.json for full results.

Evaluation (Open LLM Leaderboard v2, lm_eval)

Table with columns: Task, Metric, BF16 (base), UINT4 (this), INT4
Task	Metric	BF16 (base)	UINT4 (this)	INT4
IFEval	prompt_level_strict	0.3235	0.3383	0.3438
IFEval	inst_level_strict	0.4580	0.4664	0.4676
MATH-hard	exact_match	0.5287	0.4856	0.5083

Per-task raw metrics are shown (the six tasks use different metrics, so no single average is reported). Full result JSON and per-sample JSONL files are in eval/leaderboard/.

Evaluation (GSM8K, 5-shot)

Table with columns: Model, flexible-extract, strict-match
Model	flexible-extract	strict-match
Qwen3.6-35B-A3B BF16 (base)	0.9158 ± 0.0076	0.9090 ± 0.0079
Qwen3.6-35B-A3B RTN-INT4-W4A16	0.9045 ± 0.0081	0.8961 ± 0.0084
Qwen3.6-35B-A3B RTN-UINT4-W4A16 (this model)	0.9022 ± 0.0082	0.8976 ± 0.0083

Eval command:

bash

# 1. Create a custom task yaml that suppresses thinking (save as gsm8k_nothink/gsm8k_nothink.yaml)cat << 'EOF' > gsm8k_nothink/gsm8k_nothink.yamltask: gsm8k_nothinkdataset_path: openai/gsm8kdataset_name: mainoutput_type: generate_untiltraining_split: trainfewshot_split: traintest_split: testdoc_to_text: "Question: {{question}}\nAnswer: <think>\n\n</think>\n\n"doc_to_target: "{{answer}}"metric_list:  - metric: exact_match    aggregation: mean    higher_is_better: true    ignore_case: true    ignore_punctuation: false    regexes_to_ignore:      - ","      - "\\$"      - "(?s).*#### "      - "\\.$"generation_kwargs:  until:    - "Question:"    - "</s>"    - "<|im_end|>"  do_sample: false  temperature: 0.0repeats: 1num_fewshot: 5filter_list:  - name: "strict-match"    filter:      - function: "regex"        regex_pattern: "#### (\\-?[0-9\\.\\,]+)"      - function: "take_first"  - name: "flexible-extract"    filter:      - function: "regex"        group_select: -1        regex_pattern: "(-?[$0-9.,]{2,})|(-?[0-9]+)"      - function: "take_first"metadata:  version: 3.0EOF # 2. Run evaluationCUDA_VISIBLE_DEVICES=0 \VLLM_USE_TRITON_AWQ=1 \lm_eval run \  --model vllm \  --model_args "pretrained=hongweimeng/Qwen3.6-35B-A3B-RTN-UINT4-W4A16,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.95,enforce_eager=True" \  --tasks gsm8k_nothink \  --include_path . \  --num_fewshot 5 \  --batch_size auto

VLLM_USE_TRITON_AWQ=1 is required — the default CUDA AWQ kernel enforces OC % group_size == 0, which some MoE expert layers violate at small batch sizes. The Triton kernel has no such restriction.

vLLM inference

Requires vLLM with Quark W4A16 UINT4 support. Set VLLM_USE_TRITON_AWQ=1 to use the Triton AWQ kernel, which handles all layer shapes correctly (see GSM8K note above).

python

import osos.environ["VLLM_USE_TRITON_AWQ"] = "1" from vllm import LLM, SamplingParams llm = LLM(    model="hongweimeng/Qwen3.6-35B-A3B-RTN-UINT4-W4A16",    quantization="quark",    dtype="auto",    gpu_memory_utilization=0.95,    enforce_eager=True,)print(llm.generate(["Hello"], SamplingParams(max_tokens=32))[0].outputs[0].text)

Reproduce

This repo includes scripts/ used to quantize and evaluate:

bash

source scripts/env.shbash scripts/01_quantize_rtn_uint4.sh   # needs base model + quark + GPUbash scripts/02_eval_ppl.sh

Base model: Qwen/Qwen3.6-35B-A3B (~67 GB FP16).

Files

model.safetensors — quantized weights (~21 GB)
config.json — includes quantization_config with layer excludes
scripts/ — quantization and PPL evaluation scripts
eval/ppl_wikitext.json — Wikitext-2 PPL results
eval/leaderboard/ — Open LLM Leaderboard results (JSON + per-sample JSONL)