Qwen3.6-35B-A3B-GPTQ-w4a16 API & Inference Endpoint

Калибровочный набор данных

Источники:

nvidia/Open-SWE-Traces v1.1 — CC-BY-4.0, © NVIDIA Corporation
pyromind/agentic-tool-call-dataset-12k
ZeroAgency/ru-big-russian-dataset
mlabonne/open-perfectblend (ultrachat_200k_sft, ultrafeedback_binarized, AutoIF-instruct-61k, lmsys-arena-human-preference-55k-sharegpt)

Результирующий набор по типам:

Table with columns: Группа, Число примеров, Число токенов, Доля
Группа	Число примеров	Число токенов	Доля
Агентные трейсы	231	727,262	49.4%
Русский	212	515,638	35.0%
Английский	133	230,621	15.7%

Результаты

Контроль качества был выполнен на стандартных бенчмарках и бенчмарке MERA v2

Table with columns: Бенчмарк, BF16, W4A16-GPTQ, Recovery
Бенчмарк	BF16	W4A16-GPTQ	Recovery
GSM8K-Platinum	98.84	98.59	99.75%
MATH-500, 4-shot	98.0	98.0	100.0%
MATH-500, 0-shot	87.2	86.8	99.5%
IFEval	87.43	88.72

MERA v2

Table with columns: MERA Text 2.0, BF16, W4A16-GPTQ, Recovery
MERA Text 2.0	BF16	W4A16-GPTQ	Recovery
Aggregated	0.391	0.387	99.0%
Reasoning	0.522	0.525	100.6%
Human Centric	0.385	0.357	92.7%
Agentic	0.360	0.373

Процесс квантизации

Окружение

Оборудование: 1× NVIDIA A100-SXM4-80GB, драйвер версии 580.173.02 (CUDA 13.0).

Версии пакетов:

llmcompressor==0.13.0
compressed-tensors==0.18.0
transformers==5.14.1
torch==2.13.0+cu130

python

import json from compressed_tensors.utils import save_mtp_tensors_to_checkpointfrom datasets import Datasetfrom llmcompressor import oneshotfrom llmcompressor.modifiers.gptq import GPTQModifierfrom llmcompressor.utils import load_contextfrom transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration MODEL_ID = "Qwen/Qwen3.6-35B-A3B"SAVE_DIR = "/workspace/out/Qwen3.6-35B-A3B-W4A16-GPTQ-G128"NUM_CALIBRATION_SAMPLES = 576MAX_SEQUENCE_LENGTH = 4096 with load_context(Qwen3_5MoeForConditionalGeneration):    model = Qwen3_5MoeForConditionalGeneration.from_pretrained(        MODEL_ID,        dtype="auto",        trust_remote_code=True,    ) processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True) recipe = GPTQModifier(    targets=[        r"re:.*mlp\.experts\.\d+\.(gate_proj|up_proj|down_proj)$",        r"re:.*shared_expert\.(gate_proj|up_proj|down_proj)$",    ],    scheme="W4A16",    actorder="static",    dampening_frac=0.01,    offload_hessians=False,) with open("/workspace/calib_576.jsonl", encoding="utf-8") as f:    ds = Dataset.from_list([json.loads(line) for line in f][:NUM_CALIBRATION_SAMPLES])  def data_collator(batch):    assert len(batch) == 1    import torch    return {k: torch.tensor(v).unsqueeze(0) for k, v in batch[0].items()}  oneshot(    model=model,    recipe=recipe,    dataset=ds,    max_seq_length=MAX_SEQUENCE_LENGTH,    num_calibration_samples=NUM_CALIBRATION_SAMPLES,    moe_calibrate_all_experts=True,    pipeline="sequential",    data_collator=data_collator,) model.save_pretrained(SAVE_DIR, save_compressed=True)processor.save_pretrained(SAVE_DIR)save_mtp_tensors_to_checkpoint(source_model=MODEL_ID, dest_dir=SAVE_DIR)print("QUANTIZATION COMPLETE")

Запуск

vLLM 0.29.0, 1 × A100 80GB (SM80):

bash

vllm serve /path/to/Qwen3.6-35B-A3B-W4A16-GPTQ-G128 \  --served-model-name qwen36-35b-a3b-w4a16 \  --tensor-parallel-size 1 \  --language-model-only \  --reasoning-parser qwen3 \  --enable-auto-tool-choice \  --tool-call-parser qwen3_coder \  --max-model-len 131072 \  --gpu-memory-utilization 0.90 \  --trust-remote-code

Бенчмаркинг

Использовался lm-eval 0.4.14.dev0.

bash

lm_eval run \  --model local-chat-completions \  --model_args model=qwen36-35b-a3b-w4a16 \               base_url=http://127.0.0.1:8000/v1/chat/completions \               num_concurrent=64 max_retries=3 truncate=False tokenized_requests=False \               timeout=3600 tokenizer=Qwen/Qwen3.6-35B-A3B \  --gen_kwargs temperature=1.0 top_p=0.95 top_k=20 presence_penalty=1.5 max_gen_toks=$MAXTOK \               'until=["</s>","<|im_end|>"]' \  --apply_chat_template \  --tasks "$TASK" --seed $SEED,None,None,None \  --output_path "$OUT/${TASK}_seed${SEED}" --log_samples# run_gen_eval.sh gsm8k_platinum 64000 42 | minerva_math500 64000 42 |#           ifeval 16384 42 | mgsm_native_cot_ru 16384 42 | aime25 64000 {42,1234,...}

Бенчмарки на множественный выбор (MMLU-Pro, GPQA Diamond, MMLU-ProX ru, Global-MMLU ru) — run_mc_eval.sh TASK [LIMIT]:

bash

lm_eval run \  --model local-completions \  --model_args model=qwen36-35b-a3b-w4a16 \               base_url=http://127.0.0.1:8000/v1/completions \               num_concurrent=64 max_retries=3 truncate=False tokenized_requests=False \               max_length=8192 tokenizer=Qwen/Qwen3.6-35B-A3B \  --tasks "$TASK" --seed 42,None,None,None [--limit $LIMIT] \  --output_path "$OUT/${TASK}"# run_mc_eval.sh mmlu_pro 0 | gpqa_diamond_zeroshot 0 |#           mmlu_prox_ru 2000 | global_mmlu_full_ru 2000

Для MERA использовался официальный фреймворк MERA-Evaluation/MERA, release_v2 @ 22cfc76.

bash

vllm serve $MODEL --served-model-name qwen36-35b-a3b-w4a16 --tensor-parallel-size 1 \  --language-model-only --reasoning-parser qwen3 --max-model-len 262144 \  --gpu-memory-utilization 0.90 --trust-remote-code

Для всех задач использовался параметр max_tokens=32768, кроме SOBHard, где он был выставлен в 65536.

Группа

Число примеров

Число токенов

Доля

Агентные трейсы

231

727,262

49.4%

Русский

212

515,638

35.0%

Английский

133

230,621

15.7%

Бенчмарк

BF16

W4A16-GPTQ

Recovery

GSM8K-Platinum

98.84

98.59

99.75%

MATH-500, 4-shot

98.0

100.0%

MATH-500, 0-shot

87.2

86.8

99.5%

IFEval

87.43

88.72

MERA Text 2.0

BF16

W4A16-GPTQ

Recovery

Aggregated

0.391

0.387

99.0%

Reasoning

0.522

0.525

100.6%

Human Centric

0.385

0.357

92.7%

Agentic

0.360

0.373

import json from compressed_tensors.utils import save_mtp_tensors_to_checkpointfrom datasets import Datasetfrom llmcompressor import oneshotfrom llmcompressor.modifiers.gptq import GPTQModifierfrom llmcompressor.utils import load_contextfrom transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration MODEL_ID = "Qwen/Qwen3.6-35B-A3B"SAVE_DIR = "/workspace/out/Qwen3.6-35B-A3B-W4A16-GPTQ-G128"NUM_CALIBRATION_SAMPLES = 576MAX_SEQUENCE_LENGTH = 4096 with load_context(Qwen3_5MoeForConditionalGeneration): model = Qwen3_5MoeForConditionalGeneration.from_pretrained( MODEL_ID, dtype="auto", trust_remote_code=True, ) processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True) recipe = GPTQModifier( targets=[ r"re:.*mlp\.experts\.\d+\.(gate_proj|up_proj|down_proj)$", r"re:.*shared_expert\.(gate_proj|up_proj|down_proj)$", ], scheme="W4A16", actorder="static", dampening_frac=0.01, offload_hessians=False,) with open("/workspace/calib_576.jsonl", encoding="utf-8") as f: ds = Dataset.from_list([json.loads(line) for line in f][:NUM_CALIBRATION_SAMPLES]) def data_collator(batch): assert len(batch) == 1 import torch return {k: torch.tensor(v).unsqueeze(0) for k, v in batch[0].items()} oneshot( model=model, recipe=recipe, dataset=ds, max_seq_length=MAX_SEQUENCE_LENGTH, num_calibration_samples=NUM_CALIBRATION_SAMPLES, moe_calibrate_all_experts=True, pipeline="sequential", data_collator=data_collator,) model.save_pretrained(SAVE_DIR, save_compressed=True)processor.save_pretrained(SAVE_DIR)save_mtp_tensors_to_checkpoint(source_model=MODEL_ID, dest_dir=SAVE_DIR)print("QUANTIZATION COMPLETE")

vllm serve /path/to/Qwen3.6-35B-A3B-W4A16-GPTQ-G128 \ --served-model-name qwen36-35b-a3b-w4a16 \ --tensor-parallel-size 1 \ --language-model-only \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --max-model-len 131072 \ --gpu-memory-utilization 0.90 \ --trust-remote-code

lm_eval run \ --model local-chat-completions \ --model_args model=qwen36-35b-a3b-w4a16 \ base_url=http://127.0.0.1:8000/v1/chat/completions \ num_concurrent=64 max_retries=3 truncate=False tokenized_requests=False \ timeout=3600 tokenizer=Qwen/Qwen3.6-35B-A3B \ --gen_kwargs temperature=1.0 top_p=0.95 top_k=20 presence_penalty=1.5 max_gen_toks=$MAXTOK \ 'until=["</s>","<|im_end|>"]' \ --apply_chat_template \ --tasks "$TASK" --seed $SEED,None,None,None \ --output_path "$OUT/${TASK}_seed${SEED}" --log_samples# run_gen_eval.sh gsm8k_platinum 64000 42 | minerva_math500 64000 42 |# ifeval 16384 42 | mgsm_native_cot_ru 16384 42 | aime25 64000 {42,1234,...}

lm_eval run \ --model local-completions \ --model_args model=qwen36-35b-a3b-w4a16 \ base_url=http://127.0.0.1:8000/v1/completions \ num_concurrent=64 max_retries=3 truncate=False tokenized_requests=False \ max_length=8192 tokenizer=Qwen/Qwen3.6-35B-A3B \ --tasks "$TASK" --seed 42,None,None,None [--limit $LIMIT] \ --output_path "$OUT/${TASK}"# run_mc_eval.sh mmlu_pro 0 | gpqa_diamond_zeroshot 0 |# mmlu_prox_ru 2000 | global_mmlu_full_ru 2000

vllm serve $MODEL --served-model-name qwen36-35b-a3b-w4a16 --tensor-parallel-size 1 \ --language-model-only --reasoning-parser qwen3 --max-model-len 262144 \ --gpu-memory-utilization 0.90 --trust-remote-code

Qwen3.6-35B-A3B-GPTQ-w4a16

README

Калибровочный набор данных

Результаты

Процесс квантизации

Окружение

Запуск

Бенчмаркинг

Explore FriendliAI today

README

Калибровочный набор данных

Результаты

Процесс квантизации

Окружение

Запуск

Бенчмаркинг