GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
derikn
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
derikn
Model Tree
ibm-granite/granite-4.2-30b
Input Modalities
Output Modalities
Supported Functionality
| Base model | ibm-granite/granite-4.2-30b (Apache 2.0) |
| Parameters | 29,276,770,304 (see the note on the parameter count below) |
| Quantization | AWQ, weight-only, 4-bit (W4A16) |
| Group size | 128, symmetric, zero-point off |
| Method | AutoRound auto_awq export, gemm version |
| Kept at 16-bit | lm_head and the non-text modules |
| Size on disk | 16.4 GB in 6 safetensors shards, down from 58.6 GB at bf16 |
| Native context | 131,072 tokens, same as the base model |
| Chat template | included, with enable_thinking and low_effort control |
| Extra files | granite_thinking_parser.py, IBM's reasoning parser for vLLM |
Only the linear weights are quantized. The architecture, tokenizer, chat template and configuration match the base model, and no fine-tuning or extra training data went into this.
The Safetensors panel on this page reports a model size of about 5B parameters. That is not the size of the model. The 4-bit weights are packed into int32 tensors, eight weights per int32, and the panel's total counts that packed storage instead of the parameters it encodes.
This checkpoint holds 29,276,770,304 parameters, the same count as the base model. The Hub API's own
per-dtype breakdown adds up to exactly that figure (I32 28,454,158,336 plus BF16 822,611,968), while
the panel's single total does not. I32 and BF16 describe storage, not precision of the model: packed
4-bit weights, and the tensors held at bfloat16.
The repository mirrors the base model layout, with quantization_config.json added by AutoRound and
IBM's granite_thinking_parser.py kept alongside it.
Quantized from the bf16 checkpoint with AutoRound 0.15.1 on a single Intel Arc Pro B70 (34 GB). Weights stream to the card during tuning, which keeps VRAM bounded:
text
python layer2/quantize.py \ --model <base-model-dir> \ --format auto_awq \ --device xpu:0 \ --out-dir ./granite-4.2-30b-awq-int4 \ --low-gpu-mem --n_samples 128 --seqlen 1024 --batch_size 8 \ --torch-compileCalibration used NeelNanda/pile-10k: 128 samples of 1024 tokens each, batch size 8. lm_head and
the non-text modules stayed at 16-bit.
The script above is a private wrapper around AutoRound and is not published, so that exact command will not run for you. These are the AutoRound settings it passes, which are the whole recipe:
auto_awq, W4A16 (4-bit weight-only), group size 128, symmetric, zero-point offlm_head and the non-text modules excluded from quantizationtorch.compile enabled on the tuning stepTested on vLLM XPU (vllm/vllm-openai-xpu:nightly, vLLM 0.30.1rc1) with tensor parallelism
across two Arc Pro B70s, fp8 KV cache and the full 128K context:
bash
vllm serve /models/Granite-4.2-30B-AWQ-INT4 \ --served-model-name granite-4.2-30b-awq \ --quantization awq --dtype bfloat16 \ --max-model-len 131072 --block-size 64 \ --max-num-seqs 8 --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.90 --kv-cache-dtype fp8_e4m3 \ --enable-prefix-caching \ --tool-call-parser qwen3_coder --enable-auto-tool-choice \ --reasoning-parser granite_thinking_parser \ --trust-remote-code --tensor-parallel-size 2Container settings that matter on Arc: --device=/dev/dri, --group-add for the render and video
groups, --ipc=host, --security-opt seccomp=unconfined, and ZE_AFFINITY_MASK set to the cards you
want to use. Without the device, group and seccomp combination Level Zero reports zero devices.
Granite 4.2 has three thinking modes, selected either through OpenAI's reasoning_effort or through
the chat template. vLLM accepts the whole reasoning_effort scale, but only three states exist:
| Request | Behaviour |
|---|---|
reasoning_effort: "none" (or enable_thinking: false) | no thinking |
reasoning_effort: "low" (or low_effort: true) | brief reasoning |
omitted, or minimal / medium / high / xhigh / max | full thinking |
Measured with usage.completion_tokens_details.reasoning_tokens on the same prompt: none gives 0,
low gives about 10, the rest give between 100 and 200. minimal does not produce minimal reasoning;
it behaves like full thinking. medium, high, xhigh and max are all the same as each other.
Tested:
message.reasoning, and a LiteLLM
proxy returns it as reasoning_content.qwen3_coder parser, which matches the template format
<tool_call><function=...><parameter=...>. IBM's serving recipe recommends the same parser.Not tested:
Apache License 2.0, inherited from the base model. LICENSE and NOTICE.txt are in this repository.
python layer2/quantize.py \ --model <base-model-dir> \ --format auto_awq \ --device xpu:0 \ --out-dir ./granite-4.2-30b-awq-int4 \ --low-gpu-mem --n_samples 128 --seqlen 1024 --batch_size 8 \ --torch-compilevllm serve /models/Granite-4.2-30B-AWQ-INT4 \ --served-model-name granite-4.2-30b-awq \ --quantization awq --dtype bfloat16 \ --max-model-len 131072 --block-size 64 \ --max-num-seqs 8 --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.90 --kv-cache-dtype fp8_e4m3 \ --enable-prefix-caching \ --tool-call-parser qwen3_coder --enable-auto-tool-choice \ --reasoning-parser granite_thinking_parser \ --trust-remote-code --tensor-parallel-size 2