ronantakizawa

olmo3-32b-gptq

Deploy Dedicated

README

License: apache-2.0

Key Features

32B parameters quantized to 4-bit - 72% size reduction (64.4 GB → 18.1 GB)
Fits on a single 24 GB+ GPU for weights, instead of needing 2× 80 GB GPUs in BF16
Every decoder Linear layer quantized with GPTQ, including attention (see "Why GPTQ" below)
Fully open base model - code, data, and training details all public
vLLM compatible - compressed-tensors W4A16 (Machete kernels on H100, Marlin on Ampere/Ada)
+0.90% perplexity vs BF16 on C4 validation; MMLU 70.9% vs 71.2%, GSM8K 81.4% vs 80.8%
Quantized with multi-node distributed GPTQ - 16 H100 GPUs across 2 nodes

Model Details

Base Model: allenai/Olmo-3-1125-32B (32.2B parameters, base/pretrained model, not instruction-tuned)
Architecture: Olmo 3 (Olmo3ForCausalLM), 64 layers, hidden size 5120, grouped-query attention (40 query heads, 8 KV heads)
Quantization Method: GPTQ
Quantization Scheme: W4A16 (4-bit weights, 16-bit activations)
Group Size: 128, symmetric, static activation ordering
Calibration Dataset: C4 English (256 documents of ≥1,024 tokens, truncated to 1,024)

Size Comparison

Table with columns: Metric, Value
Metric	Value
Original (BF16)	~64.4 GB
Quantized (W4A16)	~18.1 GB
Reduction	~72%
Memory Saved	~46.3 GB

What Was Quantized

Quantized (4-bit):

All 448 decoder Linear layers: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj in all 64 layers

Preserved (BF16):

lm_head
Token embeddings
Normalization layers

How It Was Quantized: Multi-Node Distributed GPTQ

This model was quantized on Kubernetes (Amazon EKS) across two p5.48xlarge nodes (16× H100 80 GB), using LLM Compressor's distributed oneshot:

Table with columns: Step, Detail
Step	Detail
Launch	Kubernetes Indexed Job, one pod per node, gang-admitted by Kueue; `torchrun` with 8 processes per node (16 ranks)
Interconnect	NVLink inside each node; EFA between nodes via aws-ofi-nccl (all_reduce bus bandwidth 436 GB/s at 1 GB messages)
Model loading	Rank 0 loads BF16 weights to GPU and broadcasts replicas to all 16 ranks over NCCL (6 s)
Calibration	Each rank runs forward passes on its own 16-document slice (data-parallel calibration)
Compression	Per layer, Hessians are reduced to an owner rank chosen by bin packing; the owner runs GPTQ and broadcasts the quantized weights, scales, and zero points
Runtime	172 s for all 65 sequential stages; peak GPU memory 72.7 GiB (rank 0)

Multi-node note: LLM Compressor 0.14.0's get_main_device() selects the GPU by global rank, which fails on the second node (cuda:8–cuda:15 don't exist on an 8-GPU machine). The run patches it to use the local device index.

Evaluation

Tested with vLLM 0.30 on one p5.48xlarge node (H100 80 GB): this model on 1 GPU, the BF16 base model on 2 GPUs (tensor parallel) as the baseline. vLLM loads this checkpoint with MacheteLinearKernel (compressed-tensors WNA16); weights use 17.03 GiB of GPU memory.

Perplexity

C4 English validation split (not used for calibration), 100 documents × 1,024 tokens (102,300 scored tokens):

Table with columns: Model, Perplexity
Model	Perplexity
BF16 (original)	9.830
GPTQ W4A16 (this model)	9.918 (+0.90%)

Greedy generations on code, reasoning, and Japanese prompts were coherent and close to BF16 (a Fibonacci function prompt produced identical code).

Task Accuracy

lm-evaluation-harness 0.4.13 against each vLLM server (local-completions, greedy):

Table with columns: Task, Metric, BF16, GPTQ W4A16, Difference
Task	Metric	BF16	GPTQ W4A16	Difference
MMLU (0-shot, 14,042 questions)	acc	71.21%	70.91%	−0.30
ARC-Challenge (0-shot)	acc_norm	59.22%	57.51%	−1.71
HellaSwag (0-shot)	acc_norm	81.56%	81.09%	−0.47

Because both models answer the same questions, a paired test (McNemar, on the flip counts below) is the right significance check. The ARC-Challenge (p = 0.005) and HellaSwag (p < 0.001) drops are small but statistically real. MMLU (p = 0.08), GSM8K (p = 0.52), and MBPP (p = 0.72) show no significant change. MMLU is 0-shot to fit the evaluation budget; 5-shot is the more common setting.

Flips: Which Answers Changed

Similar accuracy can hide many answers that switched between right and wrong (Dutta et al., NeurIPS 2024). Per-question comparison against BF16:

Table with columns: Task, Right → wrong, Wrong → right, Flipped
Task	Right → wrong	Wrong → right	Flipped
MMLU	306	264	4.1%
ARC-Challenge	35	15	4.3%
HellaSwag	97	50	1.5%
GSM8K	74	82	11.8%
MBPP

GSM8K flips the most because a multi-step answer can diverge at any token. For scale: serving this same checkpoint with SGLang instead of vLLM flips 1.6% of GSM8K answers, so most of the 11.8% comes from quantization, not run-to-run noise.

Next-Token Agreement

The same 102,300 C4 validation positions, top-20 log-probabilities from each server:

Table with columns: Measure, Value
Measure	Value
Same top-1 token as BF16	94.97%
BF16's top-1 token within this model's top-5	99.97%
KL(BF16 ‖ GPTQ), mean / median / p99	0.0087 / 0.0034 / 0.095 nats

The KL estimate uses only BF16's top-20 tokens (87.8% of its probability mass on average) and is a lower bound for tokens missing from this model's top-20.

Throughput and Latency

vllm bench serve, random 1,024-token inputs / 256-token outputs, 200 prompts, --max-model-len 8192:

Table with columns: Model, GPUs, Request rate, Output tok/s, Median TTFT, Median TPOT
Model	GPUs	Request rate	Output tok/s	Median TTFT	Median TPOT
GPTQ W4A16	1	4/s	944	168 ms	27.9 ms
GPTQ W4A16	1	16/s	1,734	873 ms	76.2 ms
GPTQ W4A16	1	unlimited

A repeat of the GPTQ rows with --ignore-eos (every request generates exactly 256 tokens) and two runs per rate gave 944, 1,689, and 2,350 tok/s, within 3% of the table.

How to read this: the 4-bit model serves a 32B model on one GPU with 213k tokens of KV cache. At low load it matches BF16 throughput. At saturation one GPU delivers 2,294 tok/s; BF16 delivers 5,399 tok/s on two GPUs with tensor parallelism. These setups differ in GPU count and parallelism, so the gap is not a pure quantization effect. W4A16 kernels reduce weight memory traffic, not compute, so their advantage shrinks as batches grow. Choose this model to cut GPU count or fit smaller GPUs; choose BF16 on more GPUs for lowest latency under heavy load.

Usage

vLLM (recommended)

bash

vllm serve ronantakizawa/olmo3-32b-gptq --max-model-len 8192

python

from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") # Base model: use plain completions, not chatresponse = client.completions.create(    model="ronantakizawa/olmo3-32b-gptq",    prompt="The three most important ideas in distributed systems are",    max_tokens=200,)print(response.choices[0].text)

Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "ronantakizawa/olmo3-32b-gptq" tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(    model_id,    torch_dtype="auto",    device_map="auto",) inputs = tokenizer("The three most important ideas in distributed systems are", return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=200)print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Requirements

vLLM, or Transformers with compressed-tensors installed (pip install compressed-tensors)
~24 GB+ GPU VRAM (weights ~18 GB, plus KV cache)
NVIDIA GPU with compute capability 8.0+ for vLLM's W4A16 kernels (tested on H100 only)

text

## LicenseApache 2.0 (same as base model)## Citation```bibtex@misc{olmo3-32b-gptq,  title={Olmo-3-1125-32B GPTQ 4-bit (W4A16)},  author={Quantized by ronantakizawa},  year={2026},  url={https://huggingface.co/ronantakizawa/olmo3-32b-gptq}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

ronantakizawa

Model Tree

Base

allenai/Olmo-3-1125-32B

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Key Features

32B parameters quantized to 4-bit - 72% size reduction (64.4 GB → 18.1 GB)
Fits on a single 24 GB+ GPU for weights, instead of needing 2× 80 GB GPUs in BF16
Every decoder Linear layer quantized with GPTQ, including attention (see "Why GPTQ" below)
Fully open base model - code, data, and training details all public
vLLM compatible - compressed-tensors W4A16 (Machete kernels on H100, Marlin on Ampere/Ada)
+0.90% perplexity vs BF16 on C4 validation; MMLU 70.9% vs 71.2%, GSM8K 81.4% vs 80.8%
Quantized with multi-node distributed GPTQ - 16 H100 GPUs across 2 nodes

Model Details

Base Model: allenai/Olmo-3-1125-32B (32.2B parameters, base/pretrained model, not instruction-tuned)
Architecture: Olmo 3 (Olmo3ForCausalLM), 64 layers, hidden size 5120, grouped-query attention (40 query heads, 8 KV heads)
Quantization Method: GPTQ
Quantization Scheme: W4A16 (4-bit weights, 16-bit activations)
Group Size: 128, symmetric, static activation ordering
Calibration Dataset: C4 English (256 documents of ≥1,024 tokens, truncated to 1,024)

Size Comparison

Table with columns: Metric, Value
Metric	Value
Original (BF16)	~64.4 GB
Quantized (W4A16)	~18.1 GB
Reduction	~72%
Memory Saved	~46.3 GB

What Was Quantized

Quantized (4-bit):

All 448 decoder Linear layers: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj in all 64 layers

Preserved (BF16):

lm_head
Token embeddings
Normalization layers

How It Was Quantized: Multi-Node Distributed GPTQ

This model was quantized on Kubernetes (Amazon EKS) across two p5.48xlarge nodes (16× H100 80 GB), using LLM Compressor's distributed oneshot:

Table with columns: Step, Detail
Step	Detail
Launch	Kubernetes Indexed Job, one pod per node, gang-admitted by Kueue; `torchrun` with 8 processes per node (16 ranks)
Interconnect	NVLink inside each node; EFA between nodes via aws-ofi-nccl (all_reduce bus bandwidth 436 GB/s at 1 GB messages)
Model loading	Rank 0 loads BF16 weights to GPU and broadcasts replicas to all 16 ranks over NCCL (6 s)
Calibration	Each rank runs forward passes on its own 16-document slice (data-parallel calibration)
Compression	Per layer, Hessians are reduced to an owner rank chosen by bin packing; the owner runs GPTQ and broadcasts the quantized weights, scales, and zero points
Runtime	172 s for all 65 sequential stages; peak GPU memory 72.7 GiB (rank 0)

Evaluation

Perplexity

C4 English validation split (not used for calibration), 100 documents × 1,024 tokens (102,300 scored tokens):

Table with columns: Model, Perplexity
Model	Perplexity
BF16 (original)	9.830
GPTQ W4A16 (this model)	9.918 (+0.90%)

Greedy generations on code, reasoning, and Japanese prompts were coherent and close to BF16 (a Fibonacci function prompt produced identical code).

Task Accuracy

lm-evaluation-harness 0.4.13 against each vLLM server (local-completions, greedy):

Table with columns: Task, Metric, BF16, GPTQ W4A16, Difference
Task	Metric	BF16	GPTQ W4A16	Difference
MMLU (0-shot, 14,042 questions)	acc	71.21%	70.91%	−0.30
ARC-Challenge (0-shot)	acc_norm	59.22%	57.51%	−1.71
HellaSwag (0-shot)	acc_norm	81.56%	81.09%	−0.47

Flips: Which Answers Changed

Similar accuracy can hide many answers that switched between right and wrong (Dutta et al., NeurIPS 2024). Per-question comparison against BF16:

Table with columns: Task, Right → wrong, Wrong → right, Flipped
Task	Right → wrong	Wrong → right	Flipped
MMLU	306	264	4.1%
ARC-Challenge	35	15	4.3%
HellaSwag	97	50	1.5%
GSM8K	74	82	11.8%
MBPP

Next-Token Agreement

The same 102,300 C4 validation positions, top-20 log-probabilities from each server:

Table with columns: Measure, Value
Measure	Value
Same top-1 token as BF16	94.97%
BF16's top-1 token within this model's top-5	99.97%
KL(BF16 ‖ GPTQ), mean / median / p99	0.0087 / 0.0034 / 0.095 nats

The KL estimate uses only BF16's top-20 tokens (87.8% of its probability mass on average) and is a lower bound for tokens missing from this model's top-20.

Throughput and Latency

vllm bench serve, random 1,024-token inputs / 256-token outputs, 200 prompts, --max-model-len 8192:

Table with columns: Model, GPUs, Request rate, Output tok/s, Median TTFT, Median TPOT
Model	GPUs	Request rate	Output tok/s	Median TTFT	Median TPOT
GPTQ W4A16	1	4/s	944	168 ms	27.9 ms
GPTQ W4A16	1	16/s	1,734	873 ms	76.2 ms
GPTQ W4A16	1	unlimited

A repeat of the GPTQ rows with --ignore-eos (every request generates exactly 256 tokens) and two runs per rate gave 944, 1,689, and 2,350 tok/s, within 3% of the table.

Usage

vLLM (recommended)

bash

vllm serve ronantakizawa/olmo3-32b-gptq --max-model-len 8192

python

from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") # Base model: use plain completions, not chatresponse = client.completions.create(    model="ronantakizawa/olmo3-32b-gptq",    prompt="The three most important ideas in distributed systems are",    max_tokens=200,)print(response.choices[0].text)

Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "ronantakizawa/olmo3-32b-gptq" tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(    model_id,    torch_dtype="auto",    device_map="auto",) inputs = tokenizer("The three most important ideas in distributed systems are", return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=200)print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Requirements

vLLM, or Transformers with compressed-tensors installed (pip install compressed-tensors)
~24 GB+ GPU VRAM (weights ~18 GB, plus KV cache)
NVIDIA GPU with compute capability 8.0+ for vLLM's W4A16 kernels (tested on H100 only)

text

## LicenseApache 2.0 (same as base model)## Citation```bibtex@misc{olmo3-32b-gptq,  title={Olmo-3-1125-32B GPTQ 4-bit (W4A16)},  author={Quantized by ronantakizawa},  year={2026},  url={https://huggingface.co/ronantakizawa/olmo3-32b-gptq}}