GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
happyme531
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
happyme531
Model Tree
Qwen/Qwen3-4B
Input Modalities
Output Modalities
Supported Functionality
A post-training quantized derivative of Qwen/Qwen3-4B, using fixed offline rotation + QQQ with symmetric per-output-channel W4/W8 weights and dynamic per-token A8. The checkpoint retains Qwen3's native architecture, chat template, and learned Q/K normalization parameters. Its tensor payload is 2.822 GiB, and 97.73% of decoder Linear weight elements are W4.
| Property | Value |
|---|---|
| Architecture | Qwen3ForCausalLM, 36 decoder layers |
| Original source | ModelScope Qwen/Qwen3-4B |
| Source revision | 1b60d92484396d0fe2301c517823a28cc4cf2b38 |
| Quantization | Fixed offline rotation + QQQ |
| Weights | Symmetric per-channel W4/W8, FP32 stored scales |
| Weight calibration state | BF16 scale parameters and weight fake-quantization |
| Activation calibration | Dynamic per-token INT8, maxabs/127 |
| Embedding / remaining parameters | BF16 |
| Linear layers | 246 W4 / 7 W8, including W8 lm_head |
| Format | compressed-tensors pack-quantized, safetensors |
| QQQ settings | group size -1, block size 128, damping 0.01, actorder=False, per-channel MSE grid |
W8 projections: lm_head; MLP down_proj in layers 1, 6, 16;
attention v_proj in layers 24, 30, 32. Layer indices start at zero.
All other decoder Linear projections are W4. The output head is independent
from the BF16 input embedding after normalization folding.
| Weights | WikiText-2 PPL ↓ | CMMLU fixed 201 ↑ | GSM8K fixed 100 ↑ | GSM8K truncations ↓ |
|---|---|---|---|---|
| Original unquantized weights | 14.328645 | 139/201 (69.15%) | 92/100 | 0 |
| Qwen3-4B-W4A8-Mixed-QQQ-Rot | 14.971769 | 133/201 (66.17%) | 91/100 | 0 |
PPL increases by 4.49% relative
to the original weights. Both rows use the same native tokenizer and protocol.
Original BF16 weights and nonlinear operations are evaluated in FP32. Quantized
Linear layers use an INT8×INT8→INT32 reference GEMM with maxabs/127 per-token
activation quantization; NLL reduction uses FP64. TF32 is disabled.
WikiText-2 uses the original wiki.test.raw without automatically added special
tokens. All 584 complete 512-token windows are evaluated, scoring targets at
zero-based positions 257–511: 148,920 targets. CMMLU uses 3 test questions
from each of 67 subjects, seed 42, native zero-shot chat, enable_thinking=False,
and highest next-token logit among A/B/C/D. GSM8K uses a fixed 100-example test
subset, seed 42, native chat, enable_thinking=False, brief reasoning, greedy
decoding up to 512 new tokens, and numeric exact match. Truncated answers count
as incorrect. Sample IDs and protocol details are recorded in
evaluation.json.
Calibration uses 128 WikiText-2 raw training chunks of 512 tokens each,
totalling 65,536 tokens, with seed 42. Text rows are joined with two newlines
and encoded without added special tokens. The exact token offsets and dataset
revision are recorded in
quantization/calibration_selection.json.
RMSNorm weights are folded into Linear projections. The residual rotation is signed Hadamard 20×128, seed 42, and the value-head rotation is Hadamard 128. Rotations are folded offline using FP64 matrix products and stored as BF16 source weights. Learned Q/K norms are preserved, and there are no online rotation transforms. Before BF16 storage, the FP32 logit equivalence check passed with maximum absolute difference 8.6784e-5.
The reproduction preset uses BF16 weight-scale calibration state and exports
these scales as FP32. This choice reproduces the published recipe. The standalone
quantizer also supports FP32 weight calibration for your own models via
--weight-calibration-precision float32.
quantization/ provides the standalone quantizer, fixed
dependencies, exact W8 mask, and calibration reconstruction. Follow
quantization/README.md to reproduce this recipe or
quantize your own unquantized dense model with your own data and sensitive layers.
Packed-weight coverage, positive FP32 per-channel scales, model size, full stock Transformers/compressed-tensors loading, complete forward execution, 8-token cached generation, and W4/W8 decoder plus complete output-head checks passed verification. Qwen3 rotation and quantization tools are verified on a separate small Qwen3 model with learned Q/K norms and tied source embeddings.
The reported accuracy comes from the project INT32 reference runtime. Standard compressed-tensors uses its default activation scaling; benchmark parity and production throughput are unmeasured. The mixed per-channel format differs from groupwise GPTQ, AWQ, GGUF, and QQQ's native kernel checkpoint. CMMLU and GSM8K are fixed subsets rather than complete benchmark scores. Thinking-mode quality, long-context behavior, tool use, and coding after quantization are unmeasured. Floating-point differences across GPU architectures may affect regenerated weights; bitwise-identical regeneration is not claimed.
Weights and original quantization adapters are released under Apache-2.0.
The original Qwen3 model is developed by the Qwen team, Alibaba Cloud.
This is an independent quantized derivative. The solver is retrieved from
HandH1998/QQQ at the pinned revision documented
in the tools and retains its upstream terms. WikiText retains its dataset terms.
See LICENSE and NOTICE.
本模型基于 Qwen/Qwen3-4B，采用 固定离线旋转 + QQQ 进行训练后量化。权重为逐输出通道对称 W4/W8 混合位宽， 激活为逐 token 动态 A8，权重 scale 保存为 FP32。保留原始 Qwen3 架构、原生 chat 模板及学习得到的 Q/K norm 参数。
量化 tensor 体积为 2.822 GiB，decoder Linear
权重元素中 97.73% 为 W4。36 层中共有 246 个 W4 Linear、6 个 W8 decoder
Linear，另有 W8 lm_head。Embedding 和其他未量化参数保持 BF16。Checkpoint
使用 compressed-tensors pack-quantized 与 safetensors。
原始模型来自 ModelScope Qwen/Qwen3-4B，版本为
1b60d92484396d0fe2301c517823a28cc4cf2b38。QQQ 使用逐通道 MSE 网格、
group_size=-1、block_size=128、dampening_frac=0.01、actorder=False。
配方中的权重 scale 参数与 fake-quantization 使用 BF16 校准状态，导出 scale 为 FP32。
W8 范围：lm_head；第 1、6、16 层 MLP down_proj；第 24、30、32
层 attention v_proj。层号从 0 开始，其余 decoder Linear 均为 W4。归一化融合后
输出头与 BF16 输入 embedding 独立存储。
| 权重 | WikiText-2 PPL ↓ | CMMLU 固定201题 ↑ | GSM8K 固定100题 ↑ | 数学截断数 ↓ |
|---|---|---|---|---|
| 原始未量化权重 | 14.328645 | 139/201（69.15%） | 92/100 | 0 |
| 本量化版 | 14.971769 | 133/201（66.17%） | 91/100 | 0 |
相对原始权重，PPL 上升 4.49%。
两行使用相同 tokenizer 与评测协议。原始 BF16 权重与非线性操作按 FP32 计算；
量化 Linear 使用 INT8×INT8→INT32 参考 GEMM 和 maxabs/127 动态激活量化，
NLL 归约为 FP64，关闭 TF32。
PPL 使用原始 WikiText-2 wiki.test.raw，不自动添加特殊 token，评测全部 584 个
完整 512-token 窗口，对零起始位置 257–511 的 target 计分，共 148,920 target。
中文从 67 科各抽 3 道 test 题，共 201 题，seed 42，zero-shot 原生 chat，关闭
thinking，在 A/B/C/D 下一 token logit 中取最大值。数学使用 seed 42 的固定 100 道
test 题，原生 chat、关闭 thinking、简短分步作答，greedy 最多生成 512 token，
数字 exact match，截断计错。样本 ID 和协议见 evaluation.json。
校准使用 WikiText-2 raw train 128 段 × 512 token，共 65,536 token，
seed 42。原始文本行以两个换行连接，不自动添加特殊 token。固定数据集版本及
全部 token 起点见
quantization/calibration_selection.json。
RMSNorm 权重离线融合到 Linear 投影；残差使用 seed 42 的带符号 20×128 Hadamard 旋转，value-head 使用 128 维 Hadamard。矩阵运算按 FP64 离线折叠， 预处理源权重保存为 BF16。Q/K norm 保留原值，无在线旋转变换。保存 BF16 前 的 FP32 logit 等价检查通过，最大绝对差为 8.6784e-5。
复现 preset 使用本配方的 BF16 权重 scale 校准状态，导出为 FP32。量化自己的模型
时，也可通过 --weight-calibration-precision float32 使用 FP32 权重量化校准。
quantization/ 包含独立量化工具、固定依赖、精确 W8 清单及校准
重建脚本。参见 quantization/README.md，可复现本配方，
也可用自己的未量化 dense 模型、数据和敏感层清单进行量化。
Packed 权重覆盖、逐通道 FP32 正 scale、模型体积、标准 Transformers/compressed-tensors 完整加载、完整 forward、8-token 缓存生成，以及 W4/W8 decoder 和完整 lm_head 前向检查均通过。量化工具另在带有学习 Q/K norm 和原始 tied embedding 的小型 Qwen3 模型上验证了旋转和端到端量化。
表中精度来自项目 INT32 参考运行时。标准 compressed-tensors 使用其默认激活 缩放，完整基准精度与生产吞吐尚未评测。逐通道混合格式与 groupwise GPTQ、AWQ、 GGUF、QQQ 原生 kernel checkpoint 不同。中文和数学为固定子集；量化后的 thinking、 长上下文、工具调用和代码能力尚未评测。不同 GPU 架构可能影响重新量化的浮点 计算结果，不承诺逐 bit 相同。
模型权重和本仓库原创量化适配代码采用 Apache-2.0 许可。原模型由阿里云 Qwen
团队开发，本仓库为独立量化衍生版本。QQQ 求解器从固定版本上游下载，遵循其
上游条款；WikiText 遵循原数据集条款。参见 LICENSE 和 NOTICE。