ronantakizawa

seed-oss-36b-base-gptq

Deploy Dedicated

README

License: apache-2.0

Key Features

36B parameters quantized to 4-bit - 71% size reduction (72.3 GB → 21.0 GB)
Fits on a single GPU - weights use 19.9 GiB in vLLM, instead of needing 2× 80 GB GPUs in BF16
Every decoder Linear layer quantized with GPTQ, including attention
Apache 2.0 base model with a 512K-token context window
vLLM compatible - compressed-tensors W4A16 (Machete kernels on H100, Marlin on Ampere/Ada)
+1.22% perplexity vs BF16 on C4 validation; ARC-Challenge 61.4% vs 61.1%, HellaSwag 81.6% vs 82.3%, GSM8K 83.0% vs 86.7%
Quantized with multi-node distributed GPTQ - 16 H100 GPUs across 2 nodes

Model Details

Base Model: ByteDance-Seed/Seed-OSS-36B-Base (36.2B parameters, base/pretrained model, not instruction-tuned; this is the version trained with synthetic instruction data, not the -woSyn variant)
Architecture: Seed-OSS (SeedOssForCausalLM), 64 layers, hidden size 5120, grouped-query attention (80 query heads, 8 KV heads, head dim 128), MLP size 27,648, vocabulary 155,136, biases on the q/k/v projections
Quantization Method: GPTQ
Quantization Scheme: W4A16 (4-bit weights, 16-bit activations)
Group Size: 128, symmetric, static activation ordering, dampening 0.01
Calibration Dataset: C4 English (256 documents of ≥1,024 tokens, truncated to 1,024)

Size Comparison

Table with columns: Metric, Value
Metric	Value
Original (BF16)	~72.3 GB
Quantized (W4A16)	~21.0 GB
Reduction	~71%
Memory Saved	~51.3 GB

The quantized share is smaller than for models with smaller vocabularies: the 155K-token embedding table and lm_head stay in BF16 (3.2 GB of the 21.0 GB).

What Was Quantized

Quantized (4-bit):

All 448 decoder Linear layers: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj in all 64 layers

Preserved (BF16):

lm_head
Token embeddings
Normalization layers
Attention projection biases

How It Was Quantized: Multi-Node Distributed GPTQ

This model was quantized on Kubernetes (Amazon EKS, us-west-2) across two p5.48xlarge nodes (16× H100 80 GB), using LLM Compressor's distributed oneshot:

Table with columns: Step, Detail
Step	Detail
Launch	Kubernetes Indexed Job, one pod per node, gang-admitted by Kueue; `torchrun` with 8 processes per node (16 ranks)
Interconnect	NVLink inside each node; EFA between nodes via aws-ofi-nccl
Storage	BF16 weights and calibration data read from an S3 bucket mounted as a Kubernetes PersistentVolume (Mountpoint for Amazon S3 CSI driver); the checkpoint was written to a second S3-backed PersistentVolume. No pod downloaded the model
Model loading	Rank 0 reads the BF16 weights from the volume and broadcasts replicas to all 16 ranks over NCCL (240 s)
Calibration	Each rank runs forward passes on its own 16-document slice (data-parallel calibration)
Compression	Per layer, Hessians are reduced to an owner rank chosen by bin packing; the owner runs GPTQ and broadcasts the quantized weights, scales, and zero points
Runtime	179 s for all 65 sequential stages; peak GPU memory 77.1 GiB (highest across ranks)

Notes from the run:

Memory. Every rank holds a full BF16 replica (67.3 GiB), which leaves little room for GPTQ on an 80 GB H100. The embedding table and lm_head (3.0 GiB, untouched by GPTQ) were moved to each rank's own CPU memory during calibration and moved back to the GPU for saving. Peak was 77.1 GiB.
Multi-node device bug. LLM Compressor 0.14.0's get_main_device() selects the GPU by global rank, which fails on the second node (cuda:8–cuda:15 don't exist on an 8-GPU machine). The run patches it to use the local device index.
Offload across nodes. LLM Compressor rebuilds offloading around save_pretrained. Modules left on CPU get a cache shared through /dev/shm, which ranks on another node can't open. Keeping every module on the GPU before saving avoids it.

Evaluation

Tested with vLLM 0.30 on p5.48xlarge nodes (H100 80 GB): this model on 1 GPU, the BF16 base model on 2 GPUs (tensor parallel) as the baseline. vLLM loads this checkpoint with MacheteLinearKernel (compressed-tensors WNA16); weights use 19.85 GiB of GPU memory.

Perplexity

C4 English validation split (not used for calibration), 100 documents × 1,024 tokens (102,300 scored tokens):

Table with columns: Model, Perplexity
Model	Perplexity
BF16 (original)	9.010
GPTQ W4A16 (this model)	9.120 (+1.22%)

Greedy generations on code, reasoning, and Japanese prompts were coherent and close to BF16. A Fibonacci function prompt produced the same function body.

Task Accuracy

lm-evaluation-harness 0.4.13 against each vLLM server (local-completions, greedy):

Table with columns: Task, Metric, BF16, GPTQ W4A16, Difference
Task	Metric	BF16	GPTQ W4A16	Difference
ARC-Challenge (0-shot)	acc_norm	61.09%	61.43%	+0.34
HellaSwag (0-shot)	acc_norm	82.28%	81.60%	−0.68
GSM8K (5-shot)	exact_match, flexible-extract	86.66%	83.02%	−3.64
GSM8K (5-shot)	exact_match, strict	76.65%	63.61%	−13.04

Next-Token Agreement

The same 102,300 C4 validation positions, top-20 log-probabilities from each server:

Table with columns: Measure, Value
Measure	Value
Same top-1 token as BF16	94.39%
BF16's top-1 token within this model's top-5	99.94%
KL(BF16 ‖ GPTQ), mean / median / p99	0.0144 / 0.0073 / 0.129 nats

The KL estimate uses only BF16's top-20 tokens (88.8% of its probability mass on average) and is a lower bound for tokens missing from this model's top-20.

Throughput and Latency

vllm bench serve, random 1,024-token inputs / 256-token outputs, 400 prompts, --max-model-len 8192:

Table with columns: Model, GPUs, Request rate, Output tok/s, Median TTFT, Median TPOT
Model	GPUs	Request rate	Output tok/s	Median TTFT	Median TPOT
GPTQ W4A16	1	1/s	254	164 ms	15.3 ms
GPTQ W4A16	1	4/s	982	271 ms	38.9 ms
GPTQ W4A16	1	16/s	1,230	7,564 ms	119.8 ms
GPTQ W4A16	1	unlimited	1,223	15,784 ms	119.7 ms
BF16	2 (TP=2)	1/s	253	104 ms	16.6 ms
BF16	2 (TP=2)	4/s	981	110 ms	22.0 ms
BF16	2 (TP=2)	16/s	2,141	1,237 ms	72.0 ms
BF16	2 (TP=2)	unlimited	2,937	8,320 ms	47.3 ms

How to read this: one GPU running this model serves 36B with 195k tokens of KV cache. Up to 4 requests/s it matches BF16 throughput. It saturates at about 1,230 tok/s, and BF16 reaches 2,937 tok/s on two GPUs with tensor parallelism. The setups differ in GPU count and parallelism, so the gap is not a pure quantization effect. W4A16 kernels reduce weight memory traffic, not compute, so their advantage shrinks as batches grow. Choose this model to cut GPU count or fit smaller GPUs. Choose BF16 on more GPUs for lowest latency under heavy load.

The two benchmarks ran at the same time, on separate GPUs but on shared nodes, with the clients on the GPU nodes. These saturated numbers are lower than in our earlier runs of a 32B model on the same hardware. Treat them as a lower bound and repeat in isolation before relying on them.

Usage

vLLM (recommended)

bash

vllm serve ronantakizawa/seed-oss-36b-base-gptq --max-model-len 8192

python

from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") # Base model: use plain completions, not chatresponse = client.completions.create(    model="ronantakizawa/seed-oss-36b-base-gptq",    prompt="The three most important ideas in distributed systems are",    max_tokens=200,)print(response.choices[0].text)

If the weights sit on a network filesystem or an S3 mount, add --safetensors-load-strategy=eager. On a Mountpoint for S3 volume this cut load time from 190 s to 37 s, because vLLM then reads each file sequentially instead of through random mmap reads.

Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "ronantakizawa/seed-oss-36b-base-gptq" tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(    model_id,    torch_dtype="auto",    device_map="auto",) inputs = tokenizer("The three most important ideas in distributed systems are", return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=200)print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Requirements

vLLM, or Transformers ≥ 4.56 (for SeedOssForCausalLM) with compressed-tensors installed (pip install compressed-tensors)
~24 GB+ GPU VRAM (weights ~20 GiB, plus KV cache); 32 GB+ for useful context lengths
NVIDIA GPU with compute capability 8.0+ for vLLM's W4A16 kernels (tested on H100 only)

License

Apache 2.0 (same as base model)

Citation

bibtex

@misc{seed-oss-36b-base-gptq,  title={Seed-OSS-36B-Base GPTQ 4-bit (W4A16)},  author={Quantized by ronantakizawa},  year={2026},  url={https://huggingface.co/ronantakizawa/seed-oss-36b-base-gptq}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

ronantakizawa

Model Tree

Base

ByteDance-Seed/Seed-OSS-36B-Base

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Key Features

36B parameters quantized to 4-bit - 71% size reduction (72.3 GB → 21.0 GB)
Fits on a single GPU - weights use 19.9 GiB in vLLM, instead of needing 2× 80 GB GPUs in BF16
Every decoder Linear layer quantized with GPTQ, including attention
Apache 2.0 base model with a 512K-token context window
vLLM compatible - compressed-tensors W4A16 (Machete kernels on H100, Marlin on Ampere/Ada)
+1.22% perplexity vs BF16 on C4 validation; ARC-Challenge 61.4% vs 61.1%, HellaSwag 81.6% vs 82.3%, GSM8K 83.0% vs 86.7%
Quantized with multi-node distributed GPTQ - 16 H100 GPUs across 2 nodes

Model Details

Base Model: ByteDance-Seed/Seed-OSS-36B-Base (36.2B parameters, base/pretrained model, not instruction-tuned; this is the version trained with synthetic instruction data, not the -woSyn variant)
Architecture: Seed-OSS (SeedOssForCausalLM), 64 layers, hidden size 5120, grouped-query attention (80 query heads, 8 KV heads, head dim 128), MLP size 27,648, vocabulary 155,136, biases on the q/k/v projections
Quantization Method: GPTQ
Quantization Scheme: W4A16 (4-bit weights, 16-bit activations)
Group Size: 128, symmetric, static activation ordering, dampening 0.01
Calibration Dataset: C4 English (256 documents of ≥1,024 tokens, truncated to 1,024)

Size Comparison

Table with columns: Metric, Value
Metric	Value
Original (BF16)	~72.3 GB
Quantized (W4A16)	~21.0 GB
Reduction	~71%
Memory Saved	~51.3 GB

The quantized share is smaller than for models with smaller vocabularies: the 155K-token embedding table and lm_head stay in BF16 (3.2 GB of the 21.0 GB).

What Was Quantized

Quantized (4-bit):

All 448 decoder Linear layers: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj in all 64 layers

Preserved (BF16):

lm_head
Token embeddings
Normalization layers
Attention projection biases

How It Was Quantized: Multi-Node Distributed GPTQ

This model was quantized on Kubernetes (Amazon EKS, us-west-2) across two p5.48xlarge nodes (16× H100 80 GB), using LLM Compressor's distributed oneshot:

Table with columns: Step, Detail
Step	Detail
Launch	Kubernetes Indexed Job, one pod per node, gang-admitted by Kueue; `torchrun` with 8 processes per node (16 ranks)
Interconnect	NVLink inside each node; EFA between nodes via aws-ofi-nccl
Storage	BF16 weights and calibration data read from an S3 bucket mounted as a Kubernetes PersistentVolume (Mountpoint for Amazon S3 CSI driver); the checkpoint was written to a second S3-backed PersistentVolume. No pod downloaded the model
Model loading	Rank 0 reads the BF16 weights from the volume and broadcasts replicas to all 16 ranks over NCCL (240 s)
Calibration	Each rank runs forward passes on its own 16-document slice (data-parallel calibration)
Compression	Per layer, Hessians are reduced to an owner rank chosen by bin packing; the owner runs GPTQ and broadcasts the quantized weights, scales, and zero points
Runtime	179 s for all 65 sequential stages; peak GPU memory 77.1 GiB (highest across ranks)

Notes from the run:

Memory. Every rank holds a full BF16 replica (67.3 GiB), which leaves little room for GPTQ on an 80 GB H100. The embedding table and lm_head (3.0 GiB, untouched by GPTQ) were moved to each rank's own CPU memory during calibration and moved back to the GPU for saving. Peak was 77.1 GiB.
Multi-node device bug. LLM Compressor 0.14.0's get_main_device() selects the GPU by global rank, which fails on the second node (cuda:8–cuda:15 don't exist on an 8-GPU machine). The run patches it to use the local device index.
Offload across nodes. LLM Compressor rebuilds offloading around save_pretrained. Modules left on CPU get a cache shared through /dev/shm, which ranks on another node can't open. Keeping every module on the GPU before saving avoids it.

Evaluation

Perplexity

C4 English validation split (not used for calibration), 100 documents × 1,024 tokens (102,300 scored tokens):

Table with columns: Model, Perplexity
Model	Perplexity
BF16 (original)	9.010
GPTQ W4A16 (this model)	9.120 (+1.22%)

Greedy generations on code, reasoning, and Japanese prompts were coherent and close to BF16. A Fibonacci function prompt produced the same function body.

Task Accuracy

lm-evaluation-harness 0.4.13 against each vLLM server (local-completions, greedy):

Table with columns: Task, Metric, BF16, GPTQ W4A16, Difference
Task	Metric	BF16	GPTQ W4A16	Difference
ARC-Challenge (0-shot)	acc_norm	61.09%	61.43%	+0.34
HellaSwag (0-shot)	acc_norm	82.28%	81.60%	−0.68
GSM8K (5-shot)	exact_match, flexible-extract	86.66%	83.02%	−3.64
GSM8K (5-shot)	exact_match, strict	76.65%	63.61%	−13.04

Next-Token Agreement

The same 102,300 C4 validation positions, top-20 log-probabilities from each server:

Table with columns: Measure, Value
Measure	Value
Same top-1 token as BF16	94.39%
BF16's top-1 token within this model's top-5	99.94%
KL(BF16 ‖ GPTQ), mean / median / p99	0.0144 / 0.0073 / 0.129 nats

The KL estimate uses only BF16's top-20 tokens (88.8% of its probability mass on average) and is a lower bound for tokens missing from this model's top-20.

Throughput and Latency

vllm bench serve, random 1,024-token inputs / 256-token outputs, 400 prompts, --max-model-len 8192:

Table with columns: Model, GPUs, Request rate, Output tok/s, Median TTFT, Median TPOT
Model	GPUs	Request rate	Output tok/s	Median TTFT	Median TPOT
GPTQ W4A16	1	1/s	254	164 ms	15.3 ms
GPTQ W4A16	1	4/s	982	271 ms	38.9 ms
GPTQ W4A16	1	16/s	1,230	7,564 ms	119.8 ms
GPTQ W4A16	1	unlimited	1,223	15,784 ms	119.7 ms
BF16	2 (TP=2)	1/s	253	104 ms	16.6 ms
BF16	2 (TP=2)	4/s	981	110 ms	22.0 ms
BF16	2 (TP=2)	16/s	2,141	1,237 ms	72.0 ms
BF16	2 (TP=2)	unlimited	2,937	8,320 ms	47.3 ms

Usage

vLLM (recommended)

bash

vllm serve ronantakizawa/seed-oss-36b-base-gptq --max-model-len 8192

python

from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") # Base model: use plain completions, not chatresponse = client.completions.create(    model="ronantakizawa/seed-oss-36b-base-gptq",    prompt="The three most important ideas in distributed systems are",    max_tokens=200,)print(response.choices[0].text)

from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") # Base model: use plain completions, not chatresponse = client.completions.create(    model="ronantakizawa/seed-oss-36b-base-gptq",    prompt="The three most important ideas in distributed systems are",    max_tokens=200,)print(response.choices[0].text)

Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "ronantakizawa/seed-oss-36b-base-gptq" tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(    model_id,    torch_dtype="auto",    device_map="auto",) inputs = tokenizer("The three most important ideas in distributed systems are", return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=200)print(tokenizer.decode(outputs[0], skip_special_tokens=True))

from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "ronantakizawa/seed-oss-36b-base-gptq" tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(    model_id,    torch_dtype="auto",    device_map="auto",) inputs = tokenizer("The three most important ideas in distributed systems are", return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=200)print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Requirements

vLLM, or Transformers ≥ 4.56 (for SeedOssForCausalLM) with compressed-tensors installed (pip install compressed-tensors)
~24 GB+ GPU VRAM (weights ~20 GiB, plus KV cache); 32 GB+ for useful context lengths
NVIDIA GPU with compute capability 8.0+ for vLLM's W4A16 kernels (tested on H100 only)

License

Apache 2.0 (same as base model)

Citation

bibtex

@misc{seed-oss-36b-base-gptq,  title={Seed-OSS-36B-Base GPTQ 4-bit (W4A16)},  author={Quantized by ronantakizawa},  year={2026},  url={https://huggingface.co/ronantakizawa/seed-oss-36b-base-gptq}}

@misc{seed-oss-36b-base-gptq,  title={Seed-OSS-36B-Base GPTQ 4-bit (W4A16)},  author={Quantized by ronantakizawa},  year={2026},  url={https://huggingface.co/ronantakizawa/seed-oss-36b-base-gptq}}