GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
amd
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
amd
Model Tree
ibm-granite/granite-4.0-h-tiny
Input Modalities
Output Modalities
Supported Functionality
This is a quantized version of granite-4.0-h-tiny created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.
The model was quantized from granite-4.0-h-tiny using LLM Compressor with the GPTQ algorithm. This reduces the model weights from 12.9 GiB to 3.6 GiB on disk (~72% reduction).
compressed-tensors, num_bits=4, type=int, symmetric=true, group_size=128, actorder=staticpack-quantized formatHuggingFaceH4/ultrachat_200k, sequence length 2048granite-4.0-h-tiny is a hybrid Mamba-MoE model: of its 40 layers, 4 are full-attention blocks and the other 36 are Mamba (linear-attention) blocks, and every layer carries a 64-expert MoE block (top-6 routing) alongside a shared MLP.
block_sparse_moe.experts.*.{gate,up,down}_proj), the shared MLP (shared_mlp.{input,output}_linear), the Mamba projections (mamba.{in,out}_proj), and self_attn.{q,k,v,o}_proj in the 4 full-attention layers.block_sparse_moe.router), the Mamba state-space internals that are not Linear layers (conv1d, A_log, D, dt_bias, and the gated mamba.norm), lm_head, embed_tokens, and the layer norms.Two details make this work. The model is loaded inside load_context(), which is the LLM Compressor v0.13 MoE linearization path: it exposes the fused expert tensors as individual Linear submodules so GPTQ can build a Hessian per expert, with no manual module swap. And the router is skipped because it is a tiny Linear whose logits decide expert assignment, where 4-bit rounding error can flip the top-k selection and change which experts run.
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom datasets import load_dataset from llmcompressor import oneshotfrom llmcompressor.modifiers.quantization.gptq import GPTQModifierfrom llmcompressor.utils import load_context model_id = "ibm-granite/granite-4.0-h-tiny"output_dir = "./granite-4.0-h-tiny-w4a16-llmcompressor"CALIB_SIZE = 128MAX_SEQ_LENGTH = 2048 # Step 1: Load the BF16 model inside load_context(), which linearizes the MoE# experts so GPTQ can target them as ordinary Linear modules.with load_context(AutoModelForCausalLM): model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="cpu", trust_remote_code=True, )tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) # Step 2: Build the GPTQ calibration set.ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{CALIB_SIZE}]")ds = ds.map( lambda ex: {"text": "\n".join(m["content"] for m in ex["messages"] if m.get("content"))}, remove_columns=ds.column_names,)if not getattr(tokenizer, "pad_token", None): tokenizer.pad_token = tokenizer.eos_tokencalib_ds = ds.map( lambda ex: tokenizer( ex["text"], truncation=True, max_length=MAX_SEQ_LENGTH, add_special_tokens=False ), remove_columns=["text"],) # Step 3: Define the W4A16 GPTQ recipe. Experts, shared MLP, Mamba projections# and attention are all quantized; only lm_head and the MoE router are skipped.recipe = GPTQModifier( scheme="W4A16", targets=["Linear"], ignore=["lm_head", "re:.*block_sparse_moe.router"],) # Step 4: One-shot quantize with calibration data and save in# compressed-tensors format.oneshot( model=model, dataset=calib_ds, recipe=recipe, max_seq_length=MAX_SEQ_LENGTH, tokenizer=tokenizer, output_dir=output_dir, trust_remote_code_model=True,) # Smoke testinputs = tokenizer("What are we having for dinner?", return_tensors="pt")with torch.no_grad(): output = model.generate(**inputs, max_new_tokens=30)print(tokenizer.decode(output[0], skip_special_tokens=True))python
from vllm import LLM, SamplingParams model = LLM( model="amd/granite-4.0-h-tiny-w4a16-llmcompressor", dtype="bfloat16", trust_remote_code=True,) sampling_params = SamplingParams(temperature=0.7, max_tokens=256)outputs = model.generate(["Hello, how are you?"], sampling_params)print(outputs[0].outputs[0].text)text
torch==2.13.0.0zentorch==2.13.0.0vllm==0.29.0llmcompressor==0.13.0For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):
bash
# Using LLVM OpenMP (llvmopenmp)export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1) # Or using Intel OpenMP (libiomp)export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)Note: Set
LD_PRELOADbefore launching vLLM or any inference script.
The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.
| Benchmark | BF16 Baseline | W4A16 (this model) | Recovery |
|---|---|---|---|
| GSM8K (5-shot) | 0.8203 | 0.7976 | 97.23% |
bash
lm_eval \ --model vllm \ --model_args pretrained=amd/granite-4.0-h-tiny-w4a16-llmcompressor,dtype=bfloat16 \ --tasks gsm8k \ --batch_size auto \ --trust_remote_code \ --num_fewshot 5 \ --apply_chat_template \ --log_samples \ --gen_kwargs "max_gen_toks=2048" \ --output_path .conv1d, A_log, D, dt_bias, gated norms) are not Linear layers and stay in BF16, so the INT4 saving applies to the projections, experts and attention rather than to the full recurrent path.This model is distributed under the same license as the source model. See the LICENSE file for details.
Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom datasets import load_dataset from llmcompressor import oneshotfrom llmcompressor.modifiers.quantization.gptq import GPTQModifierfrom llmcompressor.utils import load_context model_id = "ibm-granite/granite-4.0-h-tiny"output_dir = "./granite-4.0-h-tiny-w4a16-llmcompressor"CALIB_SIZE = 128MAX_SEQ_LENGTH = 2048 # Step 1: Load the BF16 model inside load_context(), which linearizes the MoE# experts so GPTQ can target them as ordinary Linear modules.with load_context(AutoModelForCausalLM): model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="cpu", trust_remote_code=True, )tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) # Step 2: Build the GPTQ calibration set.ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{CALIB_SIZE}]")ds = ds.map( lambda ex: {"text": "\n".join(m["content"] for m in ex["messages"] if m.get("content"))}, remove_columns=ds.column_names,)if not getattr(tokenizer, "pad_token", None): tokenizer.pad_token = tokenizer.eos_tokencalib_ds = ds.map( lambda ex: tokenizer( ex["text"], truncation=True, max_length=MAX_SEQ_LENGTH, add_special_tokens=False ), remove_columns=["text"],) # Step 3: Define the W4A16 GPTQ recipe. Experts, shared MLP, Mamba projections# and attention are all quantized; only lm_head and the MoE router are skipped.recipe = GPTQModifier( scheme="W4A16", targets=["Linear"], ignore=["lm_head", "re:.*block_sparse_moe.router"],) # Step 4: One-shot quantize with calibration data and save in# compressed-tensors format.oneshot( model=model, dataset=calib_ds, recipe=recipe, max_seq_length=MAX_SEQ_LENGTH, tokenizer=tokenizer, output_dir=output_dir, trust_remote_code_model=True,) # Smoke testinputs = tokenizer("What are we having for dinner?", return_tensors="pt")with torch.no_grad(): output = model.generate(**inputs, max_new_tokens=30)print(tokenizer.decode(output[0], skip_special_tokens=True))from vllm import LLM, SamplingParams model = LLM( model="amd/granite-4.0-h-tiny-w4a16-llmcompressor", dtype="bfloat16", trust_remote_code=True,) sampling_params = SamplingParams(temperature=0.7, max_tokens=256)outputs = model.generate(["Hello, how are you?"], sampling_params)print(outputs[0].outputs[0].text)torch==2.13.0.0zentorch==2.13.0.0vllm==0.29.0llmcompressor==0.13.0# Using LLVM OpenMP (llvmopenmp)export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1) # Or using Intel OpenMP (libiomp)export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)lm_eval \ --model vllm \ --model_args pretrained=amd/granite-4.0-h-tiny-w4a16-llmcompressor,dtype=bfloat16 \ --tasks gsm8k \ --batch_size auto \ --trust_remote_code \ --num_fewshot 5 \ --apply_chat_template \ --log_samples \ --gen_kwargs "max_gen_toks=2048" \ --output_path .