GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
MingZwhy
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
MingZwhy
Model Tree
Qwen/Qwen3-0.6B
Input Modalities
Output Modalities
Supported Functionality
The tensors in this repo are bf16 and NOT quantized. QAT keeps high-precision master weights and applies fake-quant only inside the forward pass; what gets saved is the master copy.
python
# This gives you an UNQUANTIZED model. It will not error, and it will score higher# than the real 2.79-bit model.AutoModelForCausalLM.from_pretrained("MingZwhy/Qwen3-0.6B-W2.79A8KV8-QAT-step10000")
config.json here is a plain Qwen3ForCausalLM with no auto_map and no
edgerazor_config, so nothing in the file tells the loader to quantize. To use it as
a 2.79-bit model you must supply both the EdgeRazor code and the exact quantization
config it was trained under.
Want a model you can just load and evaluate? Use Qwen3-0.6B-W2.79A8KV16-OPD-3phase instead — quantization is baked in there.
This form is exactly what you want for further QAT/OPD, because the optimizer needs the continuous master weights. The training scripts pass the quantization config themselves:
bash
git clone https://github.com/MingZwhy/QAOPD && cd QAOPDbash tools/install_fresh_env.shSTUDENT_MODEL=<this checkpoint> bash scripts/train/run_unified_math.sh
| weights | weight_quant_uniform_symmetric_clip_per_block_mp_int1_58_int4_static_row_wise_sparse — mixed int1.58/int4, block 256, 50% high-precision → 2.79 bit effective |
| activations | int8 absmax per block (block 256) — A8 |
| KV cache | int8, block 128 (exported as KV16 for deployment) |
| embedding / lm_head | int4 absmax override |
| distillation | KD logits loss (alpha 2.0, T=1.0) + task loss (alpha 0.1), teacher Qwen3-0.6B FP |
EdgeRazor QAT distillation on general instruction data (ii/tulu/am/task mixtures with teacher-regenerated assistant turns), step 10000. Full procedure: docs/EDGERAZOR_QAD.md.
This checkpoint alone (the "warm" row) is heavily degraded; the OPD pipeline recovers it:
| Benchmark | this checkpoint | after 3-phase OPD | Qwen3-0.6B FP |
|---|---|---|---|
| GSM8K (5-shot) | 32.75 | 43.14 | 41.62 |
| MATH-500 (4-shot) | 10.40 | 24.20 | 27.20 |
| AMC23 (avg@16) | 1.25 | 4.69 | 7.81 |
Numbers measured through the deployment export, not by loading this file directly.
Apache-2.0, inherited from Qwen3-0.6B.