hongweimeng

Qwen3.6-35B-A3B-RTN-INT4-W4A16

README

License: apache-2.0

Quantization

Table with columns: Item, Value
Item	Value
Scheme	`int4_wo_128` symmetric, group size 128
Method	RTN (round-to-nearest, no calibration)
Pack	Quark `real_quantized`, `pack_method=reorder`
Quantized	MoE experts + `self_attn` + `linear_attn`
FP16 kept	`mlp.gate`, shared expert, `lm_head`, vision, MTP

Evaluation (Wikitext-2 test)

Table with columns: Metric, Value
Metric	Value
Perplexity	6.990
Tokens scored	296,815
Chunks (seq_len=2048)	145
Eval time	~249 s

See eval/ppl_wikitext.json for full results.

Evaluation (Open LLM Leaderboard, lm_eval)

Evaluated with EleutherAI/lm-evaluation-harness --tasks leaderboard on 2026-07-22. Run config: tensor_parallel_size=4, gpu_memory_utilization=0.90, max_model_len=8192, VLLM_USE_TRITON_AWQ=1.

Table with columns: Task, Metric, BF16 (base), UINT4, INT4 (this), Δ vs BF16
Task	Metric	BF16 (base)	UINT4	INT4 (this)	Δ vs BF16
leaderboard_bbh	acc_norm	0.6546	0.6270	0.6018	−5.3 pp
leaderboard_gpqa	acc_norm	0.4337	0.3658	0.3431	−9.1 pp
leaderboard_math_hard	exact_match

Full result JSON and per-sample JSONL files are in eval/leaderboard/.

Evaluation (GSM8K, 5-shot)

Evaluated alongside the BF16 base model and the UINT4 variant under identical conditions. The prompt suffix <think>\n\n</think>\n\n is prepended to each answer field to suppress the model's thinking preamble so all three runs are comparable (see custom task yaml below).

Table with columns: Model, flexible-extract, strict-match
Model	flexible-extract	strict-match
Qwen3.6-35B-A3B BF16 (base)	0.9158 ± 0.0076	0.9090 ± 0.0079
Qwen3.6-35B-A3B RTN-INT4-W4A16 (this model)	0.9045 ± 0.0081	0.8961 ± 0.0084
Qwen3.6-35B-A3B RTN-UINT4-W4A16	0.9022 ± 0.0082	0.8976 ± 0.0083

Eval command:

bash
# 1. Create a custom task yaml that suppresses thinking (save as gsm8k_nothink/gsm8k_nothink.yaml)
cat << 'EOF' > gsm8k_nothink/gsm8k_nothink.yaml
task: gsm8k_nothink
dataset_path: openai/gsm8k
dataset_name: main
output_type: generate_until
training_split: train
fewshot_split: train
test_split: test
doc_to_text: "Question: {{question}}\nAnswer: <think>\n\n</think>\n\n"
doc_to_target: "{{answer}}"
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: false
    regexes_to_ignore:
      - ","
      - "\\$"
      - "(?s).*#### "
      - "\\.$"
generation_kwargs:
  until:
    - "Question:"
    - "</s>"
    - "<|im_end|>"
  do_sample: false
  temperature: 0.0
repeats: 1
num_fewshot: 5
filter_list:
  - name: "strict-match"
    filter:
      - function: "regex"
        regex_pattern: "#### (\\-?[0-9\\.\\,]+)"
      - function: "take_first"
  - name: "flexible-extract"
    filter:
      - function: "regex"
        group_select: -1
        regex_pattern: "(-?[$0-9.,]{2,})|(-?[0-9]+)"
      - function: "take_first"
metadata:
  version: 3.0
EOF

# 2. Run evaluation
CUDA_VISIBLE_DEVICES=0 \
VLLM_USE_TRITON_AWQ=1 \
lm_eval run \
  --model vllm \
  --model_args "pretrained=hongweimeng/Qwen3.6-35B-A3B-RTN-INT4-W4A16,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.95,enforce_eager=True" \
  --tasks gsm8k_nothink \
  --include_path . \
  --num_fewshot 5 \
  --batch_size auto

VLLM_USE_TRITON_AWQ=1 is required — the default CUDA AWQ kernel enforces OC % group_size == 0, which some MoE expert layers violate at small batch sizes. The Triton kernel has no such restriction.

Comparison with UINT4

Table with columns: Scheme, Symmetric, Wikitext-2 PPL
Scheme	Symmetric	Wikitext-2 PPL
`uint4_wo_128`	No (asymmetric)	6.926
`int4_wo_128`	Yes	6.990

vLLM inference

Requires vLLM with Quark W4A16 INT4 support. Set VLLM_USE_TRITON_AWQ=1 to use the Triton AWQ kernel, which handles all layer shapes correctly (see GSM8K note above).

python
import os
os.environ["VLLM_USE_TRITON_AWQ"] = "1"

from vllm import LLM, SamplingParams

llm = LLM(
    model="hongweimeng/Qwen3.6-35B-A3B-RTN-INT4-W4A16",
    quantization="quark",
    dtype="auto",
    gpu_memory_utilization=0.95,
    enforce_eager=True,
)
print(llm.generate(["Hello"], SamplingParams(max_tokens=32))[0].outputs[0].text)

Reproduce

This repo includes scripts/ used to quantize and evaluate:

bash
source scripts/env.sh
bash scripts/01_quantize_rtn_int4.sh   # needs base model + quark + GPU
bash scripts/02_eval_ppl_int4.sh

Base model: Qwen/Qwen3.6-35B-A3B (~67 GB FP16).

Files

model.safetensors — quantized weights (~21 GB)
config.json — includes quantization_config with layer excludes
scripts/ — quantization and PPL evaluation scripts
eval/ppl_wikitext.json — Wikitext-2 PPL results
eval/leaderboard/ — Open LLM Leaderboard results (JSON + per-sample JSONL)

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

hongweimeng

Model Tree

Base

Qwen/Qwen3.6-35B-A3B

Quantized

this model

Input Modalities

Text

Image

Video

Output Modalities