GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
derikn
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
derikn
Model Tree
ibm-granite/granite-4.2-30b
Input Modalities
Output Modalities
Supported Functionality
| Base model | ibm-granite/granite-4.2-30b (Apache 2.0) |
| Parameters | 29,276,770,304 (see the note on the parameter count below) |
| Quantization | GPTQ, weight-only, 4-bit (W4A16) |
| Group size | 128, symmetric, desc_act off, damp_percent 0.01 |
| Method | AutoRound 0.15.1, auto_gptq export |
| Kept at 16-bit | lm_head and the non-text modules |
| Size on disk | 16.4 GB in 6 safetensors shards, down from 58.6 GB at bf16 |
| Native context | 131,072 tokens, same as the base model |
| Chat template | included, unchanged from the base model |
| Extra files | granite_thinking_parser.py, IBM's reasoning parser for vLLM |
Only the linear weights are quantized. The architecture, tokenizer, chat template and configuration match the base model, and no fine-tuning or extra training data went into this.
The Safetensors panel on this page reports a model size of about 5B parameters. That is not the size of the model. The 4-bit weights are packed into int32 tensors, eight weights per int32, and the panel's total counts that packed storage instead of the parameters it encodes.
This checkpoint holds 29,276,770,304 parameters, the same count as the base model. The Hub API's own
per-dtype breakdown adds up to exactly that figure, while the panel's single total does not. I32 and
BF16 describe storage, not the precision of the model: packed 4-bit weights, and the tensors held at
bfloat16.
Quantized from the bf16 checkpoint with AutoRound 0.15.1 on one Intel Arc Pro B70, with weights streamed to the card so VRAM stays bounded. The run was driven from the Quantize tab of a private dashboard, so this is the pipeline's invocation rather than a script you can run as-is:
text
python quantize.py \ --model <base-model-dir> \ --format auto_gptq --device xpu:0 \ --out-dir <output-dir> \ --low-gpu-mem --n_samples 128 --seqlen 1024 --batch_size 8 \ --torch-compileCalibration used the quantizer's default set, NeelNanda/pile-10k, at 128 samples of 1024 tokens each. The wrapper is not published, so that exact command will not run for you. These are the AutoRound settings it passes, which are the whole recipe:
auto_gptq: GPTQ, 4-bit weight-only, group size 128, symmetric, desc_act off, damp_percent 0.01lm_head and the non-text modules excluded from quantizationtorch.compile enabled on the tuning stepTested on vLLM XPU (vllm/vllm-openai-xpu:nightly, vLLM 0.30.1rc1) with tensor parallelism across
two Arc Pro B70s, an fp8 KV cache and the full 128K context:
bash
vllm serve /models/Granite-4.2-30B-GPTQ-INT4 \ --served-model-name granite-4.2-30b-gptq \ --quantization gptq --dtype bfloat16 \ --max-model-len 131072 --block-size 64 \ --max-num-seqs 8 --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.90 --kv-cache-dtype fp8_e4m3 \ --enable-prefix-caching \ --tool-call-parser qwen3_coder --enable-auto-tool-choice \ --reasoning-parser granite_thinking_parser \ --trust-remote-code --tensor-parallel-size 2Container settings that matter on Arc: --device=/dev/dri, --group-add for the render and video
groups, --ipc=host, --security-opt seccomp=unconfined, and ZE_AFFINITY_MASK set to the cards you
want to use. Without the device, group and seccomp combination Level Zero reports zero devices.
Granite 4.2 has three thinking modes, selected either through OpenAI's reasoning_effort or through
the chat template. vLLM accepts the whole reasoning_effort scale, but only three states exist:
| Request | Behaviour |
|---|---|
reasoning_effort: "none" (or enable_thinking: false) | no thinking |
reasoning_effort: "low" (or low_effort: true) | brief reasoning |
omitted, or minimal / medium / high / xhigh / max | full thinking |
Measured with usage.completion_tokens_details.reasoning_tokens on the same prompt: none gives 0,
low gives about 10, the rest give between 100 and 200. minimal does not produce minimal reasoning;
it behaves like full thinking. medium, high, xhigh and max are all the same as each other.
Tested:
message.reasoning, and a LiteLLM
proxy returns it as reasoning_content.qwen3_coder parser, which matches the template format
<tool_call><function=...><parameter=...>. IBM's serving recipe recommends the same parser.Not tested:
Apache License 2.0, inherited from the base model. LICENSE and NOTICE.txt are in this repository.
python quantize.py \ --model <base-model-dir> \ --format auto_gptq --device xpu:0 \ --out-dir <output-dir> \ --low-gpu-mem --n_samples 128 --seqlen 1024 --batch_size 8 \ --torch-compilevllm serve /models/Granite-4.2-30B-GPTQ-INT4 \ --served-model-name granite-4.2-30b-gptq \ --quantization gptq --dtype bfloat16 \ --max-model-len 131072 --block-size 64 \ --max-num-seqs 8 --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.90 --kv-cache-dtype fp8_e4m3 \ --enable-prefix-caching \ --tool-call-parser qwen3_coder --enable-auto-tool-choice \ --reasoning-parser granite_thinking_parser \ --trust-remote-code --tensor-parallel-size 2