devpramod-intel

granite-4.1-8b-quantized.w8a8

README

License: apache-2.0

Deployment with vLLM

bash
vllm serve devpramod-intel/granite-4.1-8b-quantized.w8a8 --max-model-len 32768

python
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

model_id = "devpramod-intel/granite-4.1-8b-quantized.w8a8"
tokenizer = AutoTokenizer.from_pretrained(model_id)
llm = LLM(model=model_id, max_model_len=4096)

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Who are you? Please respond in pirate speak!"}],
    tokenize=False, add_generation_prompt=True,
)
print(llm.generate(prompt, SamplingParams(temperature=0.3, max_tokens=256))[0].outputs[0].text)

Creation

bash
python quantize_w8a8_granite41.py \
  --model-dir ibm-granite/granite-4.1-8b \
  --out granite-4.1-8b-quantized.w8a8 \
  --smoothing-strength 0.8 --dampening-frac 0.1 \
  --observer mse --num-samples 512

Recipe:

yaml
quant_stage:
  quant_modifiers:
    SmoothQuantModifier:
      smoothing_strength: 0.8
      ignore: [lm_head]
      mappings:
      - - ['re:.*q_proj', 're:.*k_proj', 're:.*v_proj']
        - re:.*input_layernorm
      - - ['re:.*gate_proj', 're:.*up_proj']
        - re:.*post_attention_layernorm
      - - ['re:.*down_proj']
        - re:.*up_proj
    GPTQModifier:
      targets: [Linear]
      ignore: [lm_head]
      scheme: W8A8
      dampening_frac: 0.1
      weight_observer: mse
      sequential_targets: [GraniteDecoderLayer]

recipe.yaml in this repo is what llm-compressor actually applied and is authoritative. It additionally shows block_size: 128 and actorder: static, which are llm-compressor 0.9.0.4 defaults rather than choices — the older Granite cards predate actorder defaulting on, so this checkpoint is not bit-identical to what their recipe produced in 2025.

Calibration: neuralmagic/LLM_compression_calibration, train split, shuffle(seed=42).select(512), the dataset's raw text field with add_special_tokens=True, max_seq_length=8192.

Recipe provenance

Every knob is taken from Red Hat AI's published recipe.yaml files for the nearest architectural precedents — ibm-granite/granite-4.1-8b is a dense GraniteForCausalLM with Llama-style blocks (q/k/v + gate/up/down, RMSNorm), so the Granite 3.1 W8A8 recipes transfer directly.

Table with columns: Precedent, Relationship, Knobs it contributes
Precedent	Relationship	Knobs it contributes
RedHatAI/granite-3.1-8b-instruct-quantized.w8a8	same family, same class, same size class	`smoothing_strength=0.8`, llama mappings, `dampening_frac=0.1`, weight observer `mse`, INT8 channel-weight / token-dynamic-activation config group
RedHatAI/granite-3.1-2b-instruct-quantized.w8a8	smaller sibling	confirms the same structure at small scale (it uses 0.7 / 0.01)
RedHatAI/granite-4.1-8b-fp8

Deliberate deviations from those cards:

512 calibration samples instead of the Granite cards' 3072 — W8A8 is far less calibration-sensitive than W4A16, and 512 is the llm-compressor default.
max_seq_length=8192, not the 8196 printed on the Granite cards (a typo).
sequential_targets set to the decoder-layer class, following current Red Hat cards; it lowers peak VRAM and does not change the result.

Accuracy

No accuracy benchmark was run on this checkpoint. It exists to measure throughput and latency. The figures below are estimates by precedent, not measurements of this model, and should not be quoted as such:

Table with columns: Evidence, Measured recovery vs BF16
Evidence	Measured recovery vs BF16
`granite-3.1-8b-instruct` W8A8, identical recipe (Red Hat card)	OpenLLM v1 99.95% (70.26 vs 70.30), OpenLLM v2 98.64%, HumanEval 99.3%
`granite-3.1-2b-instruct` W8A8 (Red Hat card)	OpenLLM v1 99.52% (61.68 vs 61.98)
a granite-4.1-8b derivative quantized with this exact script (internal, 7-dataset classification basket)	aggregate ≈99.4%, 46/48 byte-identical decodes on CPU

On that basis the expected recovery here is ~99–100% on knowledge/reasoning multiple-choice suites and ~98–99% on generative suites. If you need a number you can defend, run lm-eval against both this checkpoint and the BF16 base and report the ratio.

Verification performed

config.json → quantization_config: format: int-quantized, weights num_bits 8 / channel / symmetric / observer mse, input activations num_bits 8 / token / dynamic, ignore: ["lm_head"]
all quantized weights and scales checked finite (no NaN/Inf)
checkpoint loads and generates coherent text

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

devpramod-intel

Model Tree

Base

ibm-granite/granite-4.1-8b

Quantized

this model

Input Modalities

Text

Output Modalities