malvavisc0

Qwen3.8-9B-gptq-int4

README

License: apache-2.0

Why this model

A 2.4T teacher in a 9B student. Math, code, and tool-use CoT distilled from Qwen3.8 2.4T A95B — not self-generated reasoning. MMLU (CoT) jumps +20.5 points over the Qwen3.5-9B base.
Int4 without the quality cliff. Calibrated on 256 samples that match the serving distribution — DeepSeek-R1 math traces, competitive programming solutions, and agentic tool-use conversations — rendered through the model's own chat template. Not web text. The calibration sees exactly what production traffic looks like.
Full text-stack coverage. All 200 quantizable text modules quantized (24 GatedDeltaNet + 8 full-attention layers, hybrid architecture). The vision tower is passed through untouched.
262k native context, preserved.
vLLM-ready. Ships with the extended chat template (reasoning effort control, XML tool-call format) and serves with a single vllm serve command.

Where it runs

Table with columns: Hardware, Fit, Notes
Hardware	Fit	Notes
RTX 4090 / 5090 (24 GB)	✅ comfortable	Long contexts, batch serving
RTX 3090 / A5000 (16–24 GB)	✅	The sweet spot
RTX 3080 / 4070 Ti (12–16 GB)	✅	Clamp `--max-model-len` for long contexts
Apple Silicon (16 GB+ unified)	✅	Via vLLM or transformers
NVIDIA GB10 / DGX Spark (128 GB unified)	✅	Full 262k context, no clamps
CPU-only	⚠️	Works via transformers, slow — use a GGUF instead

Rule of thumb: 8.5 GB weights + KV cache. 12 GB of total memory is the practical floor; 16 GB is comfortable; 24 GB+ lets the 262k context breathe.

Quickstart (vLLM — recommended)

bash

pip install "vllm>=0.27"vllm serve malvavisc0/Qwen3.8-9B-gptq-int4 \  --quantization gptq_marlin --max-num-seqs 10

--max-num-seqs 10 matters: this is a hybrid GatedDeltaNet/full-attention architecture, and vLLM's Mamba-style cache is happiest with a bounded batch. On 24 GB cards, add --max-model-len 32768 (or less) if you don't need the full 262k.

Then talk to it over the OpenAI API:

python

from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")resp = client.chat.completions.create(    model="malvavisc0/Qwen3.8-9B-gptq-int4",    messages=[{"role": "user", "content":        "A snail sits at the bottom of a 10 m well. Each day it climbs "        "3 m, each night it slips back 2 m. How many days to escape?"}],    temperature=0.6, top_p=0.95,    extra_body={"top_k": 20, "max_tokens": 16384},)print(resp.choices[0].message.content)

Quickstart (transformers)

python

from transformers import AutoModelForCausalLM, AutoTokenizerimport torch model_id = "malvavisc0/Qwen3.8-9B-gptq-int4"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto") messages = [{"role": "user", "content": "Prove that √2 is irrational."}]inputs = tok.apply_chat_template(    messages, add_generation_prompt=True, return_tensors="pt").to(model.device) out = model.generate(    inputs, max_new_tokens=16384,    temperature=0.6, top_p=0.95, top_k=20, do_sample=True,)print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Best practices

Sampling: temperature=0.6, top_p=0.95, top_k=20. Greedy decoding causes repetition loops in reasoning models of this class.
Token budget: be generous (max_new_tokens ≥ 8192). <think> blocks are long by design — the model deliberates, then answers. Strip the <think>...</think> span for end users.
Tool calling: native XML <tool_call> format per Qwen3.5's spec; pass tool definitions via the chat template's tools argument.
Reasoning effort: the shipped chat template supports reasoning_effort (low→xhigh) — dial it down for easy questions, up for competition math.

Quantization details

Table

Method	GPTQ Int4, group size 128, `desc_act`
Quantizer	gptqmodel 7.3.4 (torch 2.13, transformers 5.15)
Calibration	256 samples: 96 math CoT (nvidia/OpenMathReasoning), 96 code CoT (nvidia/OpenCodeReasoning), 64 agentic tool-use (nvidia/Nemotron-Agentic-v1), rendered through the shipped chat template, seq len 4096
Coverage	200/200 quantizable text modules (vision tower excluded by design — text-only calibration can't represent it)
Quality	mean per-layer GPTQ error 2.3e-5, worst 1.1e-4
Size	8.5 GB (vs 18 GB BF16)
Provenance	`aft_provenance.json` + included in this repo

Base revision: 0934f3d2327ff2df2197495278c4c46ae5a56bd9. Produced with aft (Aria Finetuner).

Sibling artifacts

empero-ai/Qwen3.8-9B — BF16 original + benchmark table
empero-ai's GGUF builds — for llama.cpp / CPU / Metal

License

Apache-2.0, inherited from the Qwen3.5-9B base via the upstream distillation. Credit to Empero for the model and the Qwen team for the architecture.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

malvavisc0

Model Tree

Base

empero-ai/Qwen3.8-9B-Distill

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Why this model

A 2.4T teacher in a 9B student. Math, code, and tool-use CoT distilled from Qwen3.8 2.4T A95B — not self-generated reasoning. MMLU (CoT) jumps +20.5 points over the Qwen3.5-9B base.
Int4 without the quality cliff. Calibrated on 256 samples that match the serving distribution — DeepSeek-R1 math traces, competitive programming solutions, and agentic tool-use conversations — rendered through the model's own chat template. Not web text. The calibration sees exactly what production traffic looks like.
Full text-stack coverage. All 200 quantizable text modules quantized (24 GatedDeltaNet + 8 full-attention layers, hybrid architecture). The vision tower is passed through untouched.
262k native context, preserved.
vLLM-ready. Ships with the extended chat template (reasoning effort control, XML tool-call format) and serves with a single vllm serve command.

Where it runs

Table with columns: Hardware, Fit, Notes
Hardware	Fit	Notes
RTX 4090 / 5090 (24 GB)	✅ comfortable	Long contexts, batch serving
RTX 3090 / A5000 (16–24 GB)	✅	The sweet spot
RTX 3080 / 4070 Ti (12–16 GB)	✅	Clamp `--max-model-len` for long contexts
Apple Silicon (16 GB+ unified)	✅	Via vLLM or transformers
NVIDIA GB10 / DGX Spark (128 GB unified)	✅	Full 262k context, no clamps
CPU-only	⚠️	Works via transformers, slow — use a GGUF instead

Rule of thumb: 8.5 GB weights + KV cache. 12 GB of total memory is the practical floor; 16 GB is comfortable; 24 GB+ lets the 262k context breathe.

Quickstart (vLLM — recommended)

bash

pip install "vllm>=0.27"vllm serve malvavisc0/Qwen3.8-9B-gptq-int4 \  --quantization gptq_marlin --max-num-seqs 10

--max-num-seqs 10 matters: this is a hybrid GatedDeltaNet/full-attention architecture, and vLLM's Mamba-style cache is happiest with a bounded batch. On 24 GB cards, add --max-model-len 32768 (or less) if you don't need the full 262k.

Then talk to it over the OpenAI API:

python

from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")resp = client.chat.completions.create(    model="malvavisc0/Qwen3.8-9B-gptq-int4",    messages=[{"role": "user", "content":        "A snail sits at the bottom of a 10 m well. Each day it climbs "        "3 m, each night it slips back 2 m. How many days to escape?"}],    temperature=0.6, top_p=0.95,    extra_body={"top_k": 20, "max_tokens": 16384},)print(resp.choices[0].message.content)

Quickstart (transformers)

python

from transformers import AutoModelForCausalLM, AutoTokenizerimport torch model_id = "malvavisc0/Qwen3.8-9B-gptq-int4"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto") messages = [{"role": "user", "content": "Prove that √2 is irrational."}]inputs = tok.apply_chat_template(    messages, add_generation_prompt=True, return_tensors="pt").to(model.device) out = model.generate(    inputs, max_new_tokens=16384,    temperature=0.6, top_p=0.95, top_k=20, do_sample=True,)print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Best practices

Sampling: temperature=0.6, top_p=0.95, top_k=20. Greedy decoding causes repetition loops in reasoning models of this class.
Token budget: be generous (max_new_tokens ≥ 8192). <think> blocks are long by design — the model deliberates, then answers. Strip the <think>...</think> span for end users.
Tool calling: native XML <tool_call> format per Qwen3.5's spec; pass tool definitions via the chat template's tools argument.
Reasoning effort: the shipped chat template supports reasoning_effort (low→xhigh) — dial it down for easy questions, up for competition math.

Quantization details

Table

Method	GPTQ Int4, group size 128, `desc_act`
Quantizer	gptqmodel 7.3.4 (torch 2.13, transformers 5.15)
Calibration	256 samples: 96 math CoT (nvidia/OpenMathReasoning), 96 code CoT (nvidia/OpenCodeReasoning), 64 agentic tool-use (nvidia/Nemotron-Agentic-v1), rendered through the shipped chat template, seq len 4096
Coverage	200/200 quantizable text modules (vision tower excluded by design — text-only calibration can't represent it)
Quality	mean per-layer GPTQ error 2.3e-5, worst 1.1e-4
Size	8.5 GB (vs 18 GB BF16)
Provenance	`aft_provenance.json` + included in this repo

Base revision: 0934f3d2327ff2df2197495278c4c46ae5a56bd9. Produced with aft (Aria Finetuner).

Sibling artifacts

empero-ai/Qwen3.8-9B — BF16 original + benchmark table
empero-ai's GGUF builds — for llama.cpp / CPU / Metal

License

Apache-2.0, inherited from the Qwen3.5-9B base via the upstream distillation. Credit to Empero for the model and the Qwen team for the architecture.