GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
moeshawky
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
moeshawky
Model Tree
Cloudflare/clef-omni
Input Modalities
Output Modalities
Supported Functionality
| Value | |
|---|---|
| Tool | auto-round 0.16.1 |
| Output format | auto_round (native safetensors), packing_format: auto_round:auto_gptq |
| Realized bits | 8 (read back from the exported config.json) |
| Group size | 128 |
| Symmetric | yes (sym: true) |
| Data type | int |
quant_method in config.json | auto-round |
| Quantized scope | thinker.model.layers (all 48 layers) |
| Weight-only exception | MoE router mlp.gate kept at 16-bit float in all 48 thinker layers |
| Effective bits/weight | 8.06 (8-bit weights + fp16 scales at group 128) |
| Tuning | iters=150, nsamples=128, seqlen=2048, batch_size=8 |
| Calibration corpus | NeelNanda/pile-10k (auto-round default), seed 42 |
| Source | Cloudflare/clef-omni, source model.safetensors.index.json SHA-256 9bd2cd0d13e8ba283e69634d6be634340915dfa1bc9abcaa5dc445f1bf5cf8fe |
| Hardware | 1x NVIDIA RTX PRO 6000 Blackwell (97,887 MiB), driver 580.82.07, CUDA 13.0 |
| Software | torch 2.11.0+cu130, transformers 5.20.1 |
| Wall clock | 21 min total (1200 s of tuning) |
Note: the exported quantization_config does not record the iteration count.
iters=150 above is the value this build was run with.
Measured from the exported safetensors headers — bytes, not estimates.
| Component | State | Size |
|---|---|---|
thinker.model.layers (48-layer MoE decoder) | 8-bit weight-only (packed int32 + fp16 scales) | 30,130,569,216 B packed + 467,140,608 B scales |
thinker.model.embed_tokens + norms | BF16 | 648,441,856 B |
thinker.lm_head | BF16 | 622,854,144 B |
thinker.visual (vision tower) | BF16 | 1,077,262,816 B |
thinker.audio_tower | BF16 | 1,295,854,336 B |
talker.model | BF16 | 6,334,582,784 B |
talker.code_predictor | BF16 | 283,140,608 B |
talker.hidden_projection / text_projection / codec_head | BF16 | 31,469,568 B |
code2wav (pre_transformer, decoder, code_embedding, upsample) | BF16 | 432,033,154 B |
joint_head.safetensors | unchanged, BF16 | 218,364,000 B |
The talker and code2wav are not used by load_release_model at all. The
vision and audio towers were deliberately left in BF16: they are outside the
model's decision path, and quantizing them would mean calibrating a
modality-specific tower against a text corpus. The MoE router gates are
retained at full precision in every layer.
| File | Size |
|---|---|
model-00001-of-00008.safetensors | 5,369,666,920 B |
model-00002-of-00008.safetensors | 5,369,153,064 B |
model-00003-of-00008.safetensors | 5,369,685,384 B |
model-00004-of-00008.safetensors | 5,369,154,848 B |
model-00005-of-00008.safetensors | 5,369,685,480 B |
model-00006-of-00008.safetensors | 5,363,558,704 B |
model-00007-of-00008.safetensors | 5,368,691,960 B |
model-00008-of-00008.safetensors | 3,752,201,498 B |
model.safetensors.index.json | 6,403,353 B |
joint_head.safetensors | 218,364,000 B |
joint_head_config.json, joint_schema_model.py | 119 B, 29,376 B |
config.json, quantization_config.json | 19,844 B, 7,578 B |
tokenizer.json, tokenizer_config.json, chat_template.jinja | 11,424,262 B, 907 B, 6,519 B |
processor_config.json, preprocessor_config.json, generation_config.json | 2,356 B, 679 B, 204 B |
LICENSE | 11,544 B |
| Total | 41,568,068,599 B (41.57 GB) |
joint_head.safetensors SHA-256:
d5ee8d7e75c9a503edce94400dda9074fc0f2f4d62729012bf0a827df947ba9c
— identical to the source release, so consumers can verify they received the
same head.
The consumer path is the upstream one: the joint schema head is loaded separately and applied to the thinker.
python
import sysfrom huggingface_hub import snapshot_download path = snapshot_download("moeshawky/clef-omni-int8-autoround")sys.path.insert(0, path)from joint_schema_model import load_release_model, systemone model, processor = load_release_model(path, device="cuda") response = systemone(model, processor, { "model": "clef-omni", "state": "Our checkout started returning errors and orders are blocked.", "questions": { "department": { "type": "choice", "instructions": "Which team should handle the message?", "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}, }, "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]}, "outage": {"type": "noul", "instructions": "Is a service down?"}, },})print(response["answers"])Loading requires auto-round and transformers>=5, because
transformers routes the auto-round quant_method to
AutoRoundQuantizer, which calls auto_round.inference.convert_model.convert_hf_model.
| Check | Result |
|---|---|
Loads via load_release_model from this directory | pass (23.9 s) |
| Consumer's own weight-integrity gate | pass — no missing, unexpected, or mismatched keys |
| Linears actually quantized (not a silent BF16 fallback) | 18,634 of 19,014 |
| Joint head SHA-256 vs source | identical |
| Top-option agreement vs the BF16 source | 15 / 15 |
| Max abs. probability delta vs BF16 | 0.0441 |
| Mean abs. probability delta vs BF16 | 0.0060 |
| Record set | 6 hand-written records, 15 questions, synthetic |
The comparison runs the same records through this build and through the unquantized BF16 release, via the same consumer path, and compares the per-option softmax probabilities.
choice, score and noul questions. It shows
the decision path works and the top option is preserved; it does not
establish task accuracy at any meaningful resolution.NeelNanda/pile-10k, a
general text corpus, while the model also has a vision and an audio tower.
Those towers were not quantized, so no audio or image data was needed, but
the calibration distribution is not the model's own decision workload.bash
auto-round --model "Cloudflare/clef-omni" \ --bits 8 --group_size 128 --iters 150 \ --nsamples 128 --seqlen 2048 --batch_size 8 \ --format "auto_round" \ --output_dir ./clef-omni-int8-autoroundApache-2.0. Derived from Cloudflare/clef-omni, which is post-trained from Qwen/Qwen3-Omni-30B-A3B-Instruct.
import sysfrom huggingface_hub import snapshot_download path = snapshot_download("moeshawky/clef-omni-int8-autoround")sys.path.insert(0, path)from joint_schema_model import load_release_model, systemone model, processor = load_release_model(path, device="cuda") response = systemone(model, processor, { "model": "clef-omni", "state": "Our checkout started returning errors and orders are blocked.", "questions": { "department": { "type": "choice", "instructions": "Which team should handle the message?", "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}, }, "urgency": {"type": "score", "criteria": ["Can wait", "This week", "Today"]}, "outage": {"type": "noul", "instructions": "Is a service down?"}, },})print(response["answers"])auto-round --model "Cloudflare/clef-omni" \ --bits 8 --group_size 128 --iters 150 \ --nsamples 128 --seqlen 2048 --batch_size 8 \ --format "auto_round" \ --output_dir ./clef-omni-int8-autoround