GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
Kaleto
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
Kaleto
Model Tree
trohrbaugh/Qwen3.5-9B-heretic-v2
Input Modalities
Output Modalities
Supported Functionality
| Base model | trohrbaugh/Qwen3.5-9B-heretic-v2 (Heretic v1.2.0 of Qwen/Qwen3.5-9B; KL 0.0793, refusals 6/100 per its card) |
| Architecture | Qwen3_5ForConditionalGeneration — 32 layers, hybrid 3× Gated DeltaNet + 1× Gated Attention per block, hidden 4096, vocab 248320, vision encoder, 262K context |
| Original size | ~19 GB (BF16) |
| Quantized size | ~11.1 GB (10.6 GB weights + 0.49 GB MTP head) |
| Quant format | NVFP4 via nvidia-modelopt 0.43.0, group_size=16, stored as modelopt checkpoint (quant_method: modelopt) |
| Quantized modules | 152 Linear layers: mlp.{gate,up,down}_proj (32 layers), self_attn.{q,k,v,o}_proj (8 full-attention layers), linear_attn.out_proj (24 DeltaNet layers) |
| Kept BF16 | lm_head, embed_tokens, entire vision tower, all norms, DeltaNet conv1d / A_log / dt_bias / all in_proj_*, MTP head |
| MTP head | 1 layer, 15 tensors, BF16, from Qwen/Qwen3.5-9B@c2022362 |
| Calibration | none (weight-only quantization, forward_loop=None; dynamic input scales) |
| Conversion date | 2026-09-16 |
Qwen3.5-9B ships with a one-layer MTP head (mtp.*, 15 tensors) that vLLM uses as a built-in draft model. trohrbaugh/Qwen3.5-9B-heretic-v2 still declares mtp_num_hidden_layers: 1 in config.json, but its safetensors index contains zero mtp.* keys: the head was dropped when the abliterated model was saved through transformers, which does not load or save MTP weights.
Without the tensors, vLLM either refuses the speculative config or drafts with an uninitialized head (≈0 % acceptance). This repo re-attaches the original head:
mtp.* tensors are copied byte-exact from Qwen/Qwen3.5-9B (revision c202236235762e1c871ad0ccb60c8ee5ba337b9a) into model-mtp.safetensorsmodel.safetensors.index.jsonmtp* is added to quantization_config.ignore (and hf_quant_config.json exclude_modules), otherwise vLLM's modelopt loader would build NVFP4 linears for the BF16 MTP layerHeretic only edits attn.o_proj and mlp.down_proj in the main decoder via directional ablation. The embeddings, lm_head and hidden-state geometry the MTP head depends on stay close to the original, and the measured acceptance rates below confirm the head still drafts well.
in_proj_* stay BF16vLLM's Qwen3.5 implementation fuses in_proj_qkv + in_proj_z into one in_proj_qkvz module (and in_proj_a + in_proj_b likewise). A fused module must have uniform precision — quantizing only part of the group makes vLLM reject the checkpoint ("some but not all shards … quantized"). The whole in_proj_* group is therefore kept BF16; only the standalone linear_attn.out_proj is quantized in DeltaNet layers.
modelopt 0.43's HF export needs three adjustments for vLLM, all applied to this checkpoint:
input_scale = 1.0 injected for every quantized linear (152×; modelopt omits them for weight-only export, vLLM otherwise registers an uninitialized parameter)config_groups.*.input_activations.dynamic = truequantization_config.ignore completed (lm_head, model.visual, model.language_model.embed_tokens, DeltaNet in_proj_* / conv1d, mtp*)vLLM 0.26.0, FlashInfer attention, FP8 KV cache. Greedy decoding, thinking disabled, 4 prompts (code, German prose, English prose, summary) × 2 rounds, 600 max tokens. seq = one request at a time, end-to-end incl. prefill. par4 = mean per-request speed with 4 concurrent requests.
| Config | seq tok/s | speedup | acceptance rate | mean accept length | par4 tok/s per request |
|---|---|---|---|---|---|
no MTP (same weights, no --speculative-config) | 23.6 | 1.00× | – | – | 25.9 |
| MTP, k=1 | 32.5 | 1.38× | 71.8 % | 1.72 |
Recommendation: num_speculative_tokens: 1. k=1 and k=2 are within noise single-stream, k=1 is fastest per request under concurrency and has the smallest KV overhead. With the MTP head reused for k>1 (only one MTP layer exists), acceptance drops with every additional draft token.
With thinking enabled and sampling (temp 0.6, top_p 0.95, top_k 20), acceptance at k=1 measured 74–88 % in two short probes.
Accuracy benchmarks (MMLU-Pro and others, NVFP4 vs. the BF16 source on the same harness) are running and will be added here.
On a single DGX Spark (GB10), vLLM 0.26.0:
Detected MTP model. Sharing target model embedding weights / lm_head weights with the draft model. and no missing-parameter warnings--tool-call-parser qwen3_xml) returns well-formed tool_calls; reasoning is split correctly by --reasoning-parser qwen3--gpu-memory-utilization 0.38 (45.9 GiB of the 128 GB UMA pool): 30.8 GiB KV cache = 1,727,305 tokens, i.e. 6.6× full 262K context. Only 8 of 32 layers (plus MTP) carry a KV cache.Vision was not re-tested after quantization. The vision tower is untouched BF16.
bash
vllm serve Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP \--speculative-config '{"method":"mtp","num_speculative_tokens":1}' \--reasoning-parser qwen3 \--enable-auto-tool-choice \--tool-call-parser qwen3_xml \--attention-backend flashinfer \--kv-cache-dtype fp8 \--max-model-len 262144 \--max-num-seqs 6 \--gpu-memory-utilization 0.38 \--enable-chunked-prefill \--max-num-batched-tokens 8192 \--enable-prefix-caching
--quantization flag needed; vLLM detects the modelopt NVFP4 config."method":"qwen3_5_mtp" still works but is deprecated in favour of "mtp".mamba_cache_mode=align for this architecture automatically. all is not supported together with MTP.--gpu-memory-utilization is a fraction of total memory. 0.38 fits 6 × 262K on a 128 GB Spark; scale it to your GPU.--speculative-config; the MTP tensors are then ignored.Qwen's recommendations for Qwen3.5-9B apply (all with top_k=20, min_p=0, repetition_penalty=1.0):
| Mode | temperature | top_p | presence_penalty |
|---|---|---|---|
| Thinking, general | 1.0 | 0.95 | 1.5 |
| Thinking, precise coding | 0.6 | 0.95 | 0.0 |
| Non-thinking, general | 0.7 | 0.8 | 1.5 |
| Non-thinking, reasoning | 1.0 | 0.95 | 1.5 |
Toggle thinking per request with chat_template_kwargs: {"enable_thinking": false}.
min_p and logit_bias have no effect while speculative decoding is active (vLLM limitation). Use top_k / top_p instead, or serve without --speculative-config if you rely on min_p.
model-00001-of-00002.safetensors, model-00002-of-00002.safetensors — NVFP4 weights + scales, BF16 vision tower / embeddings / normsmodel-mtp.safetensors — MTP head, 15 BF16 tensors (0.49 GB), unmodified from Qwen/Qwen3.5-9Bmodel.safetensors.index.json — 1231 keys incl. mtp.*config.json — upstream heretic-v2 config + quantization_config (modelopt NVFP4, ignore list incl. mtp*)hf_quant_config.json, generation_config.jsonThe base model is decensored (abliterated): it answers requests the original Qwen3.5-9B would refuse. Quantization does not add any safety layer. You are responsible for how you deploy and use it.
Apache 2.0, inherited from Qwen/Qwen3.5-9B via trohrbaugh/Qwen3.5-9B-heretic-v2.
| MTP, k=2 | 32.8 | 1.39× | 60.1 % | 2.20 | 30.1 |
| MTP, k=3 | 30.5 | 1.29× | 49.9 % | 2.50 | 29.7 |
tokenizer.json, tokenizer_config.json, vocab.json, chat_template.jinja, preprocessor_config.json, video_preprocessor_config.json — unchanged from heretic-v2