GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
happyme531
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
happyme531
Model Tree
ibm-granite/granite-4.2-3b
Input Modalities
Output Modalities
Supported Functionality
A post-training quantized derivative of ibm-granite/granite-4.2-3b,
using SmoothQuant alpha=0.3 + QQQ, mixed W4/W8 per-channel weights, and dynamic per-token A8.
The model retains the original architecture and native chat template.
The tensor payload is 2.910 GiB. 50.67% of decoder Linear weight
elements are W4; lm_head and the listed sensitive projections are W8.
| Property | Value |
|---|---|
| Architecture | GraniteForCausalLM |
| Original source | ModelScope ibm-granite/granite-4.2-3b |
| ModelScope source revision | f029229229e1c368a1e3a4245b8eba7d88939e61 |
| Quantization | SmoothQuant alpha=0.3 + QQQ |
| Weight format | Symmetric per-output-channel W4/W8, FP32 scales |
| Activation calibration | Dynamic per-token INT8, maxabs/127 |
| Embedding / remaining parameters | BF16 |
| Quantized Linear layers | 76 W4 / 205 W8 |
| Checkpoint format | compressed-tensors pack-quantized, safetensors |
| QQQ settings | group size -1, block size 128, damping 0.01, actorder=False |
W8 projections: lm_head; q/k/v/o projections in all 40 layers; all MLP projections in layers 0–9; MLP down projections in layers 26–39. Layer indices start at zero.
| Weights | WikiText-2 PPL ↓ | CMMLU fixed 201 ↑ | GSM8K fixed 100 ↑ | GSM8K truncations ↓ |
|---|---|---|---|---|
| Original unquantized weights | 15.622437 | 71/201 (35.32%) | 84/100 | 10 |
| Granite-4.2-3B-W4A8-Mixed-QQQ-SQ | 16.732086 | 76/201 (37.81%) | 83/100 | 9 |
PPL increases by 7.10% relative to the original weights. Results use
the same native tokenizer and evaluation protocol within each model family.
Original BF16 weights are evaluated with FP32 model arithmetic. Quantized Linear
layers use an INT8×INT8→INT32 reference GEMM, maxabs/127 activation quantization,
and FP32 nonlinear operations; NLL reduction uses FP64.
WikiText-2 uses the original wiki.test.raw, without added special tokens, in
complete 512-token windows. Targets at zero-based positions 257–511 are scored:
565 windows and 144,075 targets. CMMLU uses native
chat prompts and selects the highest next-token logit among A/B/C/D. GSM8K uses
greedy decoding with enable_thinking=False, a 512-new-token budget, and numeric
exact-match scoring; truncated answers count as incorrect. The math prompt asks
for brief reasoning and a final #### <number>. Sample IDs and protocol details
are recorded in evaluation.json.
The base pool contains 256 complete conversations: 128 human-verified Chinese
COIG-CQIA examples, 64 GSM8K training problems with complete solutions, and 64
SmolTalk English conversations. The SmolTalk pool contains 48
smol-magpie-ultra good/excellent examples and 16 smol-constraints examples.
Native chat rendering preserves complete assistant answers and end markers,
with a 1024-token maximum and no answer cropping.
This checkpoint uses 416 calibration observations: the 256 conversations, two additional copies of the same 64 math examples, and 32 original WikiText-2 training chunks of 512 tokens each. There are 64 distinct math problems, each with weight three. Total calibration tokens: 176,687. SmoothQuant alpha=0.3 is estimated using the same 416 observations before QQQ. Granite attention, embedding, residual, and logits multipliers are retained.
Bit widths were selected on separate WikiText validation windows, CMMLU dev335, and GSM8K train32. Calibration excludes these samples and the final CMMLU201 and GSM8K100 samples. Final subsets had also been evaluated for earlier checkpoints; the present selection uses the separate development sets.
quantization/ contains standalone quantization tools, pinned
dependencies, the exact W8 mask, and ordered calibration source references.
Follow quantization/README.md to reproduce the recipe
or quantize your own unquantized model with your own calibration data and W8 mask.
Standard Transformers/compressed-tensors full loading, forward execution, and 8-token cached generation passed verification. Packed weight coverage, positive FP32 scales, W8 source statistics, INT64 reference checks, and independent NLL and task-score reconciliation passed.
The table reports the project INT32 reference runtime. Standard compressed-tensors uses its default activation scaling; complete benchmark parity and throughput with production engines have not been measured. The mixed W4/W8 channel format is distinct from groupwise GPTQ, AWQ, GGUF, and QQQ's native kernel checkpoint. The fixed task subsets are not full benchmark results. Long-context quality, tool use, and coding after quantization have not been evaluated. Quantization reproduction across different GPU architectures can introduce floating-point differences; bitwise-identical regeneration is not claimed.
Weights and the original quantization tools are released under Apache-2.0.
The base model is developed by IBM Granite. This checkpoint is an
independent quantized derivative. QQQ solver code is retrieved from
HandH1998/QQQ at the revision documented in
the quantization tools and retains its upstream terms. Calibration data remains
subject to the source datasets' terms. See LICENSE and NOTICE.
本模型基于 ibm-granite/granite-4.2-3b，采用 SmoothQuant α=0.3 + QQQ 进行训练后量化。权重使用逐输出通道对称 W4/W8 混合位宽，激活 使用逐 token 动态 A8，权重 scale 为 FP32。保留原始架构和原生 chat 模板， embedding 及其他未量化参数保持 BF16。
量化 tensor 体积为 2.910 GiB，decoder Linear 权重元素中 50.67% 为 W4，
共 76 个 W4 / 205 个 W8 Linear。
lm_head 为 W8A8。Checkpoint 使用 compressed-tensors pack-quantized 与 safetensors。
原始模型从 ModelScope ibm-granite/granite-4.2-3b 下载，版本为
f029229229e1c368a1e3a4245b8eba7d88939e61。QQQ 参数为逐通道 group_size=-1、block_size=128、
dampening_frac=0.01、actorder=False。
W8 范围：lm_head；全部 40 层的 q/k/v/o 投影；第 0–9 层全部 MLP 投影；第 26–39 层 MLP down 投影。层号从 0 开始。
| 权重 | WikiText-2 PPL ↓ | CMMLU 固定201题 ↑ | GSM8K 固定100题 ↑ | 数学截断数 ↓ |
|---|---|---|---|---|
| 原始未量化权重 | 15.622437 | 71/201（35.32%） | 84/100 | 10 |
| 本量化版 | 16.732086 | 76/201（37.81%） | 83/100 | 9 |
相对原始权重，PPL 上升 7.10%。同模型系列使用相同 tokenizer 和
评测协议。原始 BF16 权重按 FP32 模型计算评测；量化 Linear 使用 INT8×INT8→INT32
参考 GEMM、maxabs/127 动态激活量化，非线性操作为 FP32，NLL 归约为 FP64。
PPL 使用原始 WikiText-2 wiki.test.raw，不自动添加特殊 token；按完整 512-token
窗口切分，对零起始位置 257–511 的 target 计分，共 565 个窗口、
144,075 个 target。中文选择题采用原生 chat 提示，在下一个 token
的 A/B/C/D logit 中取最大值。数学采用 greedy 解码，enable_thinking=False，最多
生成 512 个 token，按数值 exact match 计分，截断计错。数学提示要求简短推理并
以 #### <number> 结束。evaluation.json 记录样本 ID 和协议。
基础校准池为 256 条完整对话：中文 COIG-CQIA 人工核验样本 128 条、GSM8K train
完整解题 64 条、SmolTalk 英文对话 64 条。英文包括 48 条
smol-magpie-ultra good/excellent 样本和 16 条 smol-constraints 样本。
使用各自原生 chat 模板，保留完整回答和结束标记，最长 1024 token，不截断回答。
本模型实际使用 416 条加权校准观测：基础 256 条对话，加上相同 64 道数学题的 两次额外重复，以及 32 段 512-token WikiText-2 训练文本。数学共 64 道独立题，每题 统计权重为 3；校准总 token 数为 176,687。 SmoothQuant α=0.3 使用同一份 416 条加权样本重新估计，再进行 QQQ；保留 Granite 原始 attention、embedding、residual 和 logits multiplier。
位宽选择使用独立 WikiText validation、CMMLU dev335 和 GSM8K train32。校准与 这些开发样本、最终 CMMLU201/GSM8K100 分离。最终子集曾用于此前版本评测；本轮 位宽选择采用独立开发集。
quantization/ 包含独立量化脚本、固定依赖版本、精确 W8 清单及
按顺序记录的校准数据来源。参见 quantization/README.md，
可以复现本配方，也可以使用自己的未量化模型、校准数据和敏感层清单进行量化。
标准 Transformers/compressed-tensors 完整加载、forward 和 8-token 缓存生成验证 通过；packed 权重覆盖、FP32 正 scale、W8 源权重统计、INT64 整数对照、独立 NLL 与任务分数复算均通过。
表中精度来自项目 INT32 参考运行时。标准 compressed-tensors 使用库默认激活 缩放，其完整评测精度与生产推理框架吞吐尚未测量。逐通道 W4/W8 混合格式与 groupwise GPTQ、AWQ、GGUF 及 QQQ 原生 kernel checkpoint 不同。中文和数学是固定 子集结果。量化后的长上下文、工具调用和代码能力尚未评测。不同 GPU 架构重新 量化可能产生浮点差异，不承诺逐 bit 相同。
模型权重和本仓库原创量化工具采用 Apache-2.0 许可。原模型由 IBM Granite
开发，本仓库为独立量化衍生版本。QQQ 求解器从固定版本的上游仓库下载，遵循
其上游条款；校准数据遵循各原始数据集条款。参见 LICENSE 和
NOTICE。