amd

gpt-oss-20b-BF16-w8a8-llmcompressor

README

License: apache-2.0

Model Overview

Model Architecture: GptOssForCausalLM
- Input: Text
- Output: Text
Source Model: gpt-oss-20b-BF16
Supported Hardware: AMD EPYC (CPU inference)
Preferred Operating System: Linux
Inference Engine: vLLM v0.26.0
Quantization Framework: LLM Compressor v0.13.0
Quantization Method: 8-bit Weight, 8-bit Dynamic Activation Quantization (W8A8)
Compatible Stack:
- ZenDNN v6.1.0
- ZenTorch v2.11.0.3
- PyTorch v2.11.0
- LLM Compressor v0.13.0
- vLLM v0.26.0
Published with: LLM Compressor v0.13.0

This is a quantized version of gpt-oss-20b-BF16 created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from gpt-oss-20b-BF16 using LLM Compressor via the Round-to-Nearest (RTN) algorithm. This reduces the model weights from 39.0 GiB to 20.6 GiB on disk (~47% reduction).

Method: 8-bit Weight, 8-bit Dynamic Activation Quantization (W8A8)
Config: compressed-tensors, num_bits=8, type=int, symmetric=true
Weights: INT8, symmetric, per-channel (static)
Activations: INT8, symmetric, per-token (dynamic)
Kept in BF16: the MoE router (mlp.router), lm_head, embed_tokens, the attention sinks, the layer norms, and all biases. The attention projections and all 32 experts per layer are quantized.

The one non-obvious step is import gptoss_linear_experts. Stock gpt-oss stores each layer's experts as fused 3D nn.Parameter tensors, which targets=["Linear"] cannot reach — quantizing without the shim would leave the experts in BF16 and save almost nothing, since the experts are the bulk of a 20B MoE. The module registers a GptOssLinearExperts replacement that exposes them as real nn.Linear submodules, so they land in the checkpoint as per-expert gate_proj / up_proj / down_proj. The script prints the linearized class name so you can confirm the swap took effect before spending time on the quantization pass.

python
import gptoss_linear_experts  # noqa: F401  (registers GptOssLinearExperts)
from transformers import AutoModelForCausalLM, AutoTokenizer

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.utils import load_context

MODEL_ID = "unsloth/gpt-oss-20b-BF16"
SAVE_DIR = "./gpt-oss-20b-BF16-w8a8-llmcompressor"

IGNORE = [
    "lm_head",
    r"re:.*\.router$",
    r"re:.*\.router\..*",
    r"re:.*\.gate$",
    r"re:.*\.mlp\.gate$",
]

print(f"loading {MODEL_ID}", flush=True)
with load_context():
    model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

experts = model.model.layers[0].mlp.experts
print("linearized experts class:", type(experts).__name__, flush=True)

# W8A8 here is data-free (RTN), so no calibration dataset is needed.
recipe = QuantizationModifier(targets=["Linear"], scheme="W8A8", ignore=IGNORE)
oneshot(model=model, recipe=recipe)

print(f"saving to {SAVE_DIR}", flush=True)
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)
print("done", flush=True)

Quick Start

Use with vLLM

python
from vllm import LLM, SamplingParams

model = LLM(
    model="amd/gpt-oss-20b-BF16-w8a8-llmcompressor",
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

markdown
torch==2.11.0
zentorch==2.11.0.3
vllm==0.26.0
llmcompressor==0.13.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

bash
# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Table with columns: Benchmark, BF16 Baseline, W8A8 (this model), Recovery
Benchmark	BF16 Baseline	W8A8 (this model)	Recovery
GSM8K (5-shot)	0.8870	0.8772	98.90%

Evaluation Command

bash
lm_eval \
    --model vllm \
    --model_args pretrained=amd/gpt-oss-20b-BF16-w8a8-llmcompressor,dtype=bfloat16 \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --apply_chat_template \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .

Limitations

Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.11.0.3 / PyTorch v2.11.0. It may not load correctly on other versions.
CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
Linearized Experts: The experts are stored as per-expert nn.Linear weights rather than the fused 3D tensors of the source model. Loaders that expect the stock gpt-oss expert layout will not read this checkpoint.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

amd

Model Tree

Base

unsloth/gpt-oss-20b-BF16

Quantized

this model

Input Modalities

Text

Output Modalities