amd

Phi-4-mini-instruct-w4a16-llmcompressor

Model Overview

Model Architecture: Phi3ForCausalLM
- Input: Text
- Output: Text
Source Model: Phi-4-mini-instruct
Supported Hardware: AMD EPYC (CPU inference)
Preferred Operating System: Linux
Inference Engine: vLLM v0.28.0
Quantization Framework: LLM Compressor v0.12.0
Quantization Method: 4-bit Weight-Only Quantization (W4A16)
Compatible Stack:
- ZenDNN v6.1.0
- ZenTorch v2.13.0
- PyTorch v2.13.0
- LLM Compressor v0.12.0
- vLLM v0.28.0
Published with: LLM Compressor v0.12.0

This is a quantized version of Phi-4-mini-instruct created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from Phi-4-mini-instruct using LLM Compressor with the GPTQ algorithm. This reduces the model weights from 7.15 GiB to 2.69 GiB on disk (~62% reduction).

Method: 4-bit Weight-Only Quantization (W4A16)
Config: compressed-tensors, num_bits=4, type=int, symmetric=true, group_size=128, actorder=static
Weights: INT4 (4-bit integer, symmetric, group-wise), stored in pack-quantized format
Activations: BF16 (unquantized)
Group Size: 128
Calibration: 128 samples from HuggingFaceH4/ultrachat_200k, sequence length 2048
Quantized: all transformer Linear layers across the 32 layers — Phi-3 fuses its projections, so these are self_attn.qkv_proj, self_attn.o_proj, mlp.gate_up_proj, and mlp.down_proj.
, , and the layer norms.

Phi-4-mini-instruct ties its input and output embeddings, so the single BF16 embed_tokens tensor (200,064 x 3,072) doubles as lm_head and accounts for roughly 1.1 GiB of the 2.69 GiB checkpoint. The transformer body itself compresses close to the expected 4x.

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier

model_id = "microsoft/Phi-4-mini-instruct"
output_dir = "./Phi-4-mini-instruct-w4a16-llmcompressor"
CALIB_SIZE = 128
MAX_SEQ_LENGTH = 2048

# Step 1: Load the BF16 model and tokenizer.
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="cpu",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Step 2: Build the GPTQ calibration set.
ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{CALIB_SIZE}]")
ds = ds.map(
    lambda ex: {"text": "\n".join(m["content"] for m in ex["messages"] if m.get("content"))},
    remove_columns=ds.column_names,
)
if not getattr(tokenizer, "pad_token", None):
    tokenizer.pad_token = tokenizer.eos_token
calib_ds = ds.map(
    lambda ex: tokenizer(
        ex["text"], truncation=True, max_length=MAX_SEQ_LENGTH, add_special_tokens=False
    ),
    remove_columns=["text"],
)

# Step 3: Define the W4A16 GPTQ recipe. Only lm_head is skipped; this is a
# dense text-only model with no vision or audio modules to protect.
recipe = GPTQModifier(scheme="W4A16", targets="Linear", ignore=["lm_head"])

# Step 4: One-shot quantize with calibration data and save in
# compressed-tensors format.
oneshot(
    model=model,
    dataset=calib_ds,
    recipe=recipe,
    max_seq_length=MAX_SEQ_LENGTH,
    tokenizer=tokenizer,
    output_dir=output_dir,
    trust_remote_code_model=True,
)

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Quick Start

Use with vLLM

python
from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Phi-4-mini-instruct-w4a16-llmcompressor",
    dtype="bfloat16",
    trust_remote_code=True,
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

markdown
torch==2.13.0
zentorch==2.13.0
vllm==0.28.0
llmcompressor==0.12.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

bash
# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Table with columns: Benchmark, BF16 Baseline, W4A16 (this model), Recovery
Benchmark	BF16 Baseline	W4A16 (this model)	Recovery
GSM8K (5-shot)	0.8158	0.7657	93.86%

Evaluation Command

bash
lm_eval \
    --model vllm \
    --model_args pretrained=amd/Phi-4-mini-instruct-w4a16-llmcompressor,dtype=bfloat16 \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --apply_chat_template \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .

Limitations

Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.13.0 / PyTorch v2.13.0. It may not load correctly on other versions.
CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
Accuracy Trade-off: At 3.8B parameters this is a small model, and INT4 weight-only quantization costs about 5 points of GSM8K accuracy (93.86% recovery). If accuracy matters more than footprint, prefer the W8A8 variant.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

amd

Model Tree

Base

microsoft/Phi-4-mini-instruct

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Model Overview

Model Architecture: Phi3ForCausalLM
- Input: Text
- Output: Text
Source Model: Phi-4-mini-instruct
Supported Hardware: AMD EPYC (CPU inference)
Preferred Operating System: Linux
Inference Engine: vLLM v0.28.0
Quantization Framework: LLM Compressor v0.12.0
Quantization Method: 4-bit Weight-Only Quantization (W4A16)
Compatible Stack:
- ZenDNN v6.1.0
- ZenTorch v2.13.0
- PyTorch v2.13.0
- LLM Compressor v0.12.0
- vLLM v0.28.0
Published with: LLM Compressor v0.12.0

This is a quantized version of Phi-4-mini-instruct created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from Phi-4-mini-instruct using LLM Compressor with the GPTQ algorithm. This reduces the model weights from 7.15 GiB to 2.69 GiB on disk (~62% reduction).

Method: 4-bit Weight-Only Quantization (W4A16)
Config: compressed-tensors, num_bits=4, type=int, symmetric=true, group_size=128, actorder=static
Weights: INT4 (4-bit integer, symmetric, group-wise), stored in pack-quantized format
Activations: BF16 (unquantized)
Group Size: 128
Calibration: 128 samples from HuggingFaceH4/ultrachat_200k, sequence length 2048
Quantized: all transformer Linear layers across the 32 layers — Phi-3 fuses its projections, so these are self_attn.qkv_proj, self_attn.o_proj, mlp.gate_up_proj, and mlp.down_proj.
, , and the layer norms.

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier

model_id = "microsoft/Phi-4-mini-instruct"
output_dir = "./Phi-4-mini-instruct-w4a16-llmcompressor"
CALIB_SIZE = 128
MAX_SEQ_LENGTH = 2048

# Step 1: Load the BF16 model and tokenizer.
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="cpu",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Step 2: Build the GPTQ calibration set.
ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{CALIB_SIZE}]")
ds = ds.map(
    lambda ex: {"text": "\n".join(m["content"] for m in ex["messages"] if m.get("content"))},
    remove_columns=ds.column_names,
)
if not getattr(tokenizer, "pad_token", None):
    tokenizer.pad_token = tokenizer.eos_token
calib_ds = ds.map(
    lambda ex: tokenizer(
        ex["text"], truncation=True, max_length=MAX_SEQ_LENGTH, add_special_tokens=False
    ),
    remove_columns=["text"],
)

# Step 3: Define the W4A16 GPTQ recipe. Only lm_head is skipped; this is a
# dense text-only model with no vision or audio modules to protect.
recipe = GPTQModifier(scheme="W4A16", targets="Linear", ignore=["lm_head"])

# Step 4: One-shot quantize with calibration data and save in
# compressed-tensors format.
oneshot(
    model=model,
    dataset=calib_ds,
    recipe=recipe,
    max_seq_length=MAX_SEQ_LENGTH,
    tokenizer=tokenizer,
    output_dir=output_dir,
    trust_remote_code_model=True,
)

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Quick Start

Use with vLLM

python
from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Phi-4-mini-instruct-w4a16-llmcompressor",
    dtype="bfloat16",
    trust_remote_code=True,
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

markdown
torch==2.13.0
zentorch==2.13.0
vllm==0.28.0
llmcompressor==0.12.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

bash
# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Table with columns: Benchmark, BF16 Baseline, W4A16 (this model), Recovery
Benchmark	BF16 Baseline	W4A16 (this model)	Recovery
GSM8K (5-shot)	0.8158	0.7657	93.86%

Evaluation Command

bash
lm_eval \
    --model vllm \
    --model_args pretrained=amd/Phi-4-mini-instruct-w4a16-llmcompressor,dtype=bfloat16 \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --apply_chat_template \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .

Limitations

Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.13.0 / PyTorch v2.13.0. It may not load correctly on other versions.
CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
Accuracy Trade-off: At 3.8B parameters this is a small model, and INT4 weight-only quantization costs about 5 points of GSM8K accuracy (93.86% recovery). If accuracy matters more than footprint, prefer the W8A8 variant.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Phi-4-mini-instruct-w4a16-llmcompressor

README

Model Overview

Quantization

Quick Start

Use with vLLM

Requirements

OpenMP Setup

Evaluation

Evaluation Command

Limitations

License

Explore FriendliAI today

README

Model Overview

Quantization

Quick Start

Use with vLLM

Requirements

OpenMP Setup

Evaluation

Evaluation Command

Limitations

License