GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
junuke
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
junuke
Model Tree
Qwen/Qwen3-0.6B
Input Modalities
Output Modalities
Supported Functionality
model.safetensors stores the codes and zero-points packed two per byte, so plain
AutoModelForCausalLM.from_pretrained cannot read it; load_realquant_model rebuilds the
RealQuantLinear modules and unpacks them.
python
import torchfrom huggingface_hub import snapshot_downloadfrom quantlinear import load_realquant_model # https://github.com/junuke/MYPTQ local = snapshot_download("junuke/Qwen3-0.6B-MYPTQ-w4g128")model, tokenizer = load_realquant_model(local, dtype=torch.float16, device="cuda") ids = tokenizer("The capital of France is", return_tensors="pt").input_ids.cuda()print(tokenizer.decode(model(input_ids=ids).logits[0, -1].argmax())) # ' Paris'The ONNX graph keeps its weights INT4-packed and needs ONNX Runtime >= 1.16. Its tensors live in
onnx/model.onnx.data, which must sit next to onnx/model.onnx. Download the folder with
local_dir=: the default cache stores files as symlinks into its blob store, and onnxruntime rejects
external data whose real path leaves the model directory.
python
import numpy as np, onnxruntime as ort graph = snapshot_download("junuke/Qwen3-0.6B-MYPTQ-w4g128", allow_patterns=["onnx/*"], local_dir="onnx_model")sess = ort.InferenceSession(f"{graph}/onnx/model.onnx", providers=["CUDAExecutionProvider", "CPUExecutionProvider"])logits = sess.run(["logits"], {"input_ids": ids.cpu().numpy().astype(np.int64)})[0]Activations are fp16. The graph takes input_ids [batch, seq], returns logits [batch, seq, vocab],
and builds its causal mask internally (no KV cache input).
| Model | Body weights | MMLU acc | vs baseline | C4 ppl |
|---|---|---|---|---|
| Qwen3-0.6B baseline | fp16 | 0.4000 | - | 32.51 |
| naive W4 (RTN g128, min/max, data-free) | 4-bit | 0.3483 | -5.17%p (-12.9%) | 39.51 (+21.5%) |
| SmoothQuant init + W4 (untrained) | 4-bit | 0.3524 | -4.76%p (-11.9%) | 36.45 (+12.1%) |
| MYPTQ (smooth + compensation + clip, CE+KD trained) | 4-bit | 0.4294 | +2.94%p (+7.3%) | 32.89 (+1.2%) |
| ONNX graph | Size | SNR (16 x 512 tokens, wikitext2 test) |
|---|---|---|
| fp32 graph (exactness check) | 820.1 MB | 117.9 dB (min 116.8), top-1 100.00% |
| fp16 graph (deliverable) | 516.5 MB | 52.6 dB (min 51.8), top-1 99.61% |
| Qwen3-0.6B (bf16) | MYPTQ W4 g128 | ratio | |
|---|---|---|---|
| Body weights (28 layers; the assignment's compression metric) | 840.1 MB | 218.3 MB (4-bit codes + fp16 scale + 4-bit zero, 4.16 bit/weight) | 3.85x |
| Embedding (+ tied lm_head, excluded, kept 16-bit) | 296.8 MB | 296.8 MB | 1.00x |
| Total | 1136.9 MB | 515.1 MB | 2.21x (45.3% of original) |
| Files on disk | - | HF safetensors 515.2 MB / ONNX fp16 516.5 MB |
| Metric | Value |
|---|---|
| MMLU (baseline -> MYPTQ) | 0.4000 -> 0.4294 (+2.94%p (+7.3%)) |
| C4 perplexity (baseline -> MYPTQ) | 32.51 -> 32.89 (+1.2%) |
| Body compression | 3.85x (840 -> 218 MB) |
| Total model size | 45.3% of original |
| ONNX SNR fp32 / fp16 graph | 117.9 dB / 52.6 dB |
import torchfrom huggingface_hub import snapshot_downloadfrom quantlinear import load_realquant_model # https://github.com/junuke/MYPTQ local = snapshot_download("junuke/Qwen3-0.6B-MYPTQ-w4g128")model, tokenizer = load_realquant_model(local, dtype=torch.float16, device="cuda") ids = tokenizer("The capital of France is", return_tensors="pt").input_ids.cuda()print(tokenizer.decode(model(input_ids=ids).logits[0, -1].argmax())) # ' Paris'import numpy as np, onnxruntime as ort graph = snapshot_download("junuke/Qwen3-0.6B-MYPTQ-w4g128", allow_patterns=["onnx/*"], local_dir="onnx_model")sess = ort.InferenceSession(f"{graph}/onnx/model.onnx", providers=["CUDAExecutionProvider", "CPUExecutionProvider"])logits = sess.run(["logits"], {"input_ids": ids.cpu().numpy().astype(np.int64)})[0]