syedazeez

VibeThinker-3B-W4A16-G128

The story

I wanted to run VibeThinker-3B, a small but strong math/STEM reasoning model, on a laptop with an RTX 3050 (6 GB VRAM, 50 W). The problem:

Table with columns: Format, Weights, Fits 6 GB?
Format	Weights	Fits 6 GB?
BF16 (original)	~5.8 GB	❌ weights alone fill the card; nothing left for KV cache + overhead
W4A16 (this repo)	~2.0 GB	✅ leaves ~2.5 GB for KV cache (≈74k tokens)

So I quantized it locally, then loaded the result back into vLLM and confirmed it produces correct, fully-reasoned answers — all on the 6 GB laptop GPU.

What was done

Method: GPTQ, W4A16 scheme (4-bit symmetric INT weights, FP16 activations), group size 128, lm_head left unquantized.
Tool: llmcompressor 0.12, output in compressed-tensors (pack-quantized) format.
Calibration: 512 samples from HuggingFaceH4/ultrachat_200k @ 2048 tokens.
Hardware used for quantization: RTX 3050 6 GB — the model was loaded on CPU and quantized layer-by-layer (sequential GPU on-loading), keeping peak VRAM ~3.2 GB.

Measured results (RTX 3050 6 GB, vLLM 0.22)

Table with columns: Metric, Value
Metric	Value
Weights on GPU	1.99 GiB
KV cache available	2.55 GiB (74,144 tokens)
Max concurrency @ 8192 ctx	9.05×
Throughput	~67 tok/s (eager)
Kernel	Marlin W4A16 + FlashAttention 2

Benchmarks

Both models below are 4-bit, ~2 GB, running on a single RTX 3050 6 GB laptop GPU. Evaluated with lm-evaluation-harness (vLLM backend), 50-sample subset, greedy decode.

Benchmark comparison

Table with columns: Benchmark (metric), VibeThinker-3B-W4A16, Qwen2.5-3B-Instruct-AWQ
Benchmark (metric)	VibeThinker-3B-W4A16	Qwen2.5-3B-Instruct-AWQ
GSM8K (flexible-extract, 5-shot)	84%	70%
ARC-Challenge (acc_norm, 0-shot)	44%	46%

VibeThinker (the math/reasoning specialist) clearly leads on GSM8K math, while the two are comparable on ARC-Challenge general reasoning — the expected "specialist vs generalist" pattern at equal size/footprint.

Notes: 50-sample subset (±~6% stderr), indicative not publishable. ARC is run as 0-shot loglikelihood multiple-choice, which under-represents reasoning models (and is depressed by chat-template wrapping for both), so treat the ARC row as a rough wash. GSM8K is generative and the more meaningful signal here.

Usage (vLLM)

bash

vllm serve syedazeez/VibeThinker-3B-W4A16-G128 \  --max-model-len 8192 \  --gpu-memory-utilization 0.92 \  --dtype float16

python

from vllm import LLM, SamplingParams llm = LLM(model="syedazeez/VibeThinker-3B-W4A16-G128",          max_model_len=8192, gpu_memory_utilization=0.92, dtype="float16")out = llm.chat(    [[{"role": "user", "content": "What is the remainder when 7^100 is divided by 13?"}]],    SamplingParams(temperature=0.6, max_tokens=1500),)print(out[0].outputs[0].text)

This is a reasoning model: it emits a <think>...</think> trace before the final answer, so give it a generous max_tokens (≥1500) or the answer may be truncated.

Notes & limitations

Quantization is lossy; on hard benchmarks expect a small quality drop vs the BF16 original. For most math/STEM prompts the reasoning quality is preserved in informal testing.
Inherits the base model's scope: tuned for verifiable tasks (competition math, coding, STEM). Not trained for tool-calling/agents or broad open-domain knowledge.
KV cache fits ~74k tokens at this config, so --max-model-len can be pushed well above 8192 on 6 GB.

License & attribution

This derivative is released under the MIT License, the same license as the original WeiboAI/VibeThinker-3B. All credit for the model itself goes to the original authors. Only the 4-bit quantization was performed here.

bibtex

@misc{xu2026vibethinker3bexploringfrontierverifiable,      title={VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models},      author={Sen Xu and Shixi Liu and Wei Wang and Jixin Min and Yingwei Dai and Zhibin Yin and Yirong Chen and Xin Zhou and Junlin Zhang},      year={2026},      eprint={2606.16140},      archivePrefix={arXiv},      primaryClass={cs.AI},      url={https://arxiv.org/abs/2606.16140}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

syedazeez

Model Tree

Base

WeiboAI/VibeThinker-3B

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

The story

I wanted to run VibeThinker-3B, a small but strong math/STEM reasoning model, on a laptop with an RTX 3050 (6 GB VRAM, 50 W). The problem:

Table with columns: Format, Weights, Fits 6 GB?
Format	Weights	Fits 6 GB?
BF16 (original)	~5.8 GB	❌ weights alone fill the card; nothing left for KV cache + overhead
W4A16 (this repo)	~2.0 GB	✅ leaves ~2.5 GB for KV cache (≈74k tokens)

So I quantized it locally, then loaded the result back into vLLM and confirmed it produces correct, fully-reasoned answers — all on the 6 GB laptop GPU.

What was done

Method: GPTQ, W4A16 scheme (4-bit symmetric INT weights, FP16 activations), group size 128, lm_head left unquantized.
Tool: llmcompressor 0.12, output in compressed-tensors (pack-quantized) format.
Calibration: 512 samples from HuggingFaceH4/ultrachat_200k @ 2048 tokens.
Hardware used for quantization: RTX 3050 6 GB — the model was loaded on CPU and quantized layer-by-layer (sequential GPU on-loading), keeping peak VRAM ~3.2 GB.

Measured results (RTX 3050 6 GB, vLLM 0.22)

Table with columns: Metric, Value
Metric	Value
Weights on GPU	1.99 GiB
KV cache available	2.55 GiB (74,144 tokens)
Max concurrency @ 8192 ctx	9.05×
Throughput	~67 tok/s (eager)
Kernel	Marlin W4A16 + FlashAttention 2

Benchmarks

Both models below are 4-bit, ~2 GB, running on a single RTX 3050 6 GB laptop GPU. Evaluated with lm-evaluation-harness (vLLM backend), 50-sample subset, greedy decode.

Benchmark comparison

Table with columns: Benchmark (metric), VibeThinker-3B-W4A16, Qwen2.5-3B-Instruct-AWQ
Benchmark (metric)	VibeThinker-3B-W4A16	Qwen2.5-3B-Instruct-AWQ
GSM8K (flexible-extract, 5-shot)	84%	70%
ARC-Challenge (acc_norm, 0-shot)	44%	46%

Notes: 50-sample subset (±~6% stderr), indicative not publishable. ARC is run as 0-shot loglikelihood multiple-choice, which under-represents reasoning models (and is depressed by chat-template wrapping for both), so treat the ARC row as a rough wash. GSM8K is generative and the more meaningful signal here.

Usage (vLLM)

bash

vllm serve syedazeez/VibeThinker-3B-W4A16-G128 \  --max-model-len 8192 \  --gpu-memory-utilization 0.92 \  --dtype float16

vllm serve syedazeez/VibeThinker-3B-W4A16-G128 \  --max-model-len 8192 \  --gpu-memory-utilization 0.92 \  --dtype float16

python

from vllm import LLM, SamplingParams llm = LLM(model="syedazeez/VibeThinker-3B-W4A16-G128",          max_model_len=8192, gpu_memory_utilization=0.92, dtype="float16")out = llm.chat(    [[{"role": "user", "content": "What is the remainder when 7^100 is divided by 13?"}]],    SamplingParams(temperature=0.6, max_tokens=1500),)print(out[0].outputs[0].text)

from vllm import LLM, SamplingParams llm = LLM(model="syedazeez/VibeThinker-3B-W4A16-G128",          max_model_len=8192, gpu_memory_utilization=0.92, dtype="float16")out = llm.chat(    [[{"role": "user", "content": "What is the remainder when 7^100 is divided by 13?"}]],    SamplingParams(temperature=0.6, max_tokens=1500),)print(out[0].outputs[0].text)

This is a reasoning model: it emits a <think>...</think> trace before the final answer, so give it a generous max_tokens (≥1500) or the answer may be truncated.

Notes & limitations

Quantization is lossy; on hard benchmarks expect a small quality drop vs the BF16 original. For most math/STEM prompts the reasoning quality is preserved in informal testing.
Inherits the base model's scope: tuned for verifiable tasks (competition math, coding, STEM). Not trained for tool-calling/agents or broad open-domain knowledge.
KV cache fits ~74k tokens at this config, so --max-model-len can be pushed well above 8192 on 6 GB.

License & attribution

bibtex

@misc{xu2026vibethinker3bexploringfrontierverifiable,      title={VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models},      author={Sen Xu and Shixi Liu and Wei Wang and Jixin Min and Yingwei Dai and Zhibin Yin and Yirong Chen and Xin Zhou and Junlin Zhang},      year={2026},      eprint={2606.16140},      archivePrefix={arXiv},      primaryClass={cs.AI},      url={https://arxiv.org/abs/2606.16140}}

@misc{xu2026vibethinker3bexploringfrontierverifiable,      title={VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models},      author={Sen Xu and Shixi Liu and Wei Wang and Jixin Min and Yingwei Dai and Zhibin Yin and Yirong Chen and Xin Zhou and Junlin Zhang},      year={2026},      eprint={2606.16140},      archivePrefix={arXiv},      primaryClass={cs.AI},      url={https://arxiv.org/abs/2606.16140}}

VibeThinker-3B-W4A16-G128

README

The story

What was done

Measured results (RTX 3050 6 GB, vLLM 0.22)

Benchmarks

Usage (vLLM)

Notes & limitations

License & attribution

Explore FriendliAI today

README

The story

What was done

Measured results (RTX 3050 6 GB, vLLM 0.22)

Benchmarks

Usage (vLLM)

Notes & limitations

License & attribution