GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
malvavisc0
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
malvavisc0
Model Tree
empero-ai/Qwable-9B-Claude-Fable-5
Input Modalities
Output Modalities
Supported Functionality
| Original model | empero-ai/Qwable-9B-Claude-Fable-5 |
| Base architecture | Qwen3_5ForConditionalGeneration (Qwen3.5-9B, multimodal: image-text-to-text) |
| Quant method | GPTQ, 4-bit, symmetric, group size 128, desc_act=false |
| Quantizer | gptqmodel 7.3.4 |
| Calibration | FineWeb-Edu, 256 samples × 2048 tokens, chat template applied |
| On-disk size | ~8.1 GB (3 safetensors shards) |
| Quantized modules | 200 (text backbone: attention + MLP, all 32 layers) |
| Avg / max GPTQ loss | 9.4e-5 / 4.7e-4 |
GPTQ is applied only to the text backbone's linear layers. The following remain in bf16 (standard and recommended — they are small and/or accuracy-critical):
lm_head (lm_head=false),embed_tokens (untied, 248,320 × 4096 — the largest single unquantized block),That is why an "Int4 9B" model is ~8.1 GB rather than ~4.5 GB: roughly half the bytes are unquantized embedding tables. If you only need text inference, this is still a large VRAM win over the bf16 original (~18 GB) and runs on a single 12 GB GPU with offloading.
Quantized module classes (per layer, all 32 layers covered):
linear_attn.in_proj_qkv, linear_attn.in_proj_z, linear_attn.out_proj (Gated DeltaNet linear-attention layers)self_attn.q_proj, self_attn.k_proj, self_attn.v_proj, self_attn.o_proj (full-attention layers, every 4th)mlp.gate_proj, mlp.up_proj, mlp.down_projjson
{"bits": 4,"group_size": 128,"desc_act": false,"sym": true,"lm_head": false,"method": "gptq","pack_dtype": "int32","meta": {"quantizer": ["gptqmodel:7.3.4"],"uri": "https://github.com/modelcloud/gptqmodel","damp_percent": 0.05,"damp_auto_increment": 0.01,"true_sequential": true,"static_groups": false,"act_group_aware": true,"mse": 0.0,"fallback": { "strategy": "rtn", "threshold": "0.5%", "smooth": null },"pack_impl": "cpu","gc_mode": "interval"}}
| Dataset | fineweb-edu |
| Samples | 256 |
| Sequence length | 2048 |
| Chat template applied | yes |
⚠️ Calibration caveat. The base model is a coding/agentic/reasoning distill trained on traces up to ~74k tokens. This quantization was calibrated on general educational web text at 2048 tokens — it does not match the model's specialty domain and does not exercise long context. Expect the largest (still small, given the loss figures below) quality deltas on heavy coding tasks and very long inputs compared to the bf16 original. A coding/agent-trace calibration set at longer
seq_lenwould likely improve this further.
quant_log.csv)| value | |
|---|---|
| Modules | 200 |
| Average loss | 9.40e-5 |
| Max loss | 4.72e-4 (layer 28, linear_attn.in_proj_qkv) |
Losses are uniformly tiny across all layers — no module tripped the 0.5% RTN fallback threshold. This
indicates a clean, well-conditioned quantization with no outlier layers.
The architecture is multimodal (Qwen3_5ForConditionalGeneration). Load it the same way as the original —
GPTQ weights are picked up automatically from quantization_config. You need
gptqmodel (or a recent transformers + optimum GPTQ backend)
and the Qwen3.5 kernels (flash-linear-attention + a CUDA-matched causal_conv1d), otherwise the
linear-attention layers fall back to slow PyTorch ops.
python
import torchfrom transformers import AutoModelForImageTextToText, AutoTokenizermodel_id = "malvavisc0/Qwable-9B-Claude-Fable-5-GPTQ-Int4"tok = AutoTokenizer.from_pretrained(model_id)model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="bfloat16", device_map="auto",)messages = [{"role": "user", "content": "Write a Python function that merges two sorted lists."}]text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tok(text, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=2048, do_sample=True,temperature=0.7, top_p=0.95, top_k=20, repetition_penalty=1.05,)# Output starts with a <!thinking>...</thinking> reasoning block, then the final answer.print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Sampling notes (inherited from the original card): the model is a reasoning model — every response
opens with a <!thinking> block; parse/strip it for end users. repetition_penalty≈1.05 prevents rare
non-terminating reasoning loops; allow generous max_new_tokens.
For vLLM / ExLlamaV serving, prefer their native GPTQ loader and a Marlin kernel for best throughput.
rope_parameters in config.json (rope_theta: 10000.0, plain default rope).
This field is not present in the original model and was injected during quantization; it conflicts with
the correct text_config.rope_parameters (rope_theta: 10000000, mrope with section [11, 11, 10]).
In practice the text sub-config takes precedence, but loaders that read the top-level field may emit warnings
or apply the wrong RoPE. If you see odd long-context behavior, delete the top-level rope_parameters from
config.json before loading.quantization_config.meta.offload_to_disk_path contains a leftover local scratch path
(/tmp/gptqmodel_cfwi2by5) from the build host. It is metadata only and has no effect at load time.json
{"source_model": "empero-ai/Qwable-9B-Claude-Fable-5","quantization": { "format": "gptq", "bits": 4, "group_size": 128, "desc_act": false },"calibration": { "dataset": "fineweb-edu", "samples": 256, "seq_len": 2048, "chat_template_applied": true },"versions": { "aft": "0.0.1", "python": "3.14.6", "torch": "2.13.0+cu130","transformers": "5.14.1", "gptqmodel": "7.3.4" }}
apache-2.0, inherited from the base Qwen3.5-9B weights and the original fine-tune. The fine-tuning data
originates from generated traces of Claude Fable 5 and GPT-5.5 — see the
original card's Provenance & licensing section
for the third-party-terms caveat that applies to downstream commercial use.
empero-ai/Qwable-9B-Claude-Fable-5 (Empero)gptqmodel by ModelCloud