GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
maci0
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
maci0
Model Tree
bottlecapai/ThinkingCap-Qwen3.6-27B
Input Modalities
Output Modalities
Supported Functionality
ThinkingCap ships a multi-token prediction (MTP) head (mtp_num_hidden_layers: 1) — a small
extra module that predicts the next token(s) so the model can self-speculate: it drafts ahead
and verifies in one pass, cutting decode latency without a separate draft model.
That head is on the critical path for acceptance rate. If it were quantized to 4-bit, its draft
predictions would drift from what the full-precision main model would have produced, the verifier
would reject more drafts, and the speculative speedup would shrink — you would pay the MTP compute
and get less of the latency back. So the quant recipe ignores re:.*mtp.* and leaves the head
at bf16 (it is tiny relative to the 27B backbone, so the size cost is negligible). The 4-bit main
model does the heavy lifting; the full-precision MTP head keeps acceptance high.
Near-lossless versus the bf16 source, 18 GB vs ~55 GB bf16 (~33%), at wikitext-2 perplexity 7.25 and KL divergence 0.0408 to the original. GPTQ error compensation and an MSE observer keep the drop from bf16 minimal; the header lists the full characteristics and Quantization covers the recipe.
NVFP4 is auto-detected from config.json (compressed-tensors); no quantization flag
needed. --reasoning-parser qwen3 splits the block into .
<think>reasoning_contentbash
vllm serve maci0/ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 \ --served-model-name thinkingcap-27b-abliterated-nvfp4 \ --max-model-len 131072 \ --gpu-memory-utilization 0.90 \ --kv-cache-dtype fp8 \ --reasoning-parser qwen3--max-model-len 131072 is a safe default; raise it if memory allows.--language-model-only to skip the vision tower and free KV cache for text use.The bf16 MTP head enables vLLM's built-in self-speculative decoding. Enable it with a speculative config that uses the model's own next-token-prediction module (no separate draft model):
bash
vllm serve maci0/ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 \ --served-model-name thinkingcap-27b-abliterated-nvfp4 \ --max-model-len 131072 --kv-cache-dtype fp8 --reasoning-parser qwen3 \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'num_speculative_tokens matches the head's depth (mtp_num_hidden_layers: 1). Measure acceptance
rate on your traffic; because the head is bf16, acceptance is not penalized by quantization.
python
from openai import OpenAIclient = OpenAI(base_url="http://localhost:8000/v1", api_key="x")r = client.chat.completions.create( model="thinkingcap-27b-abliterated-nvfp4", messages=[{"role": "user", "content": "Explain why a heap is better than a sorted array for a priority queue."}],)print(r.choices[0].message.content)A 27B Qwen3.5-family vision-language model (ThinkingCap-Qwen3.6-27B) with thinking-mode reasoning, a 256K context window, and an MTP head for speculative decoding.
mtp_num_hidden_layers: 1) for multi-token / speculative decoding.max_position_embeddings 262144).Heretic removes the refusal direction with a TPE-optimized
search over per-component ablation strength, jointly minimizing refusal rate and KL divergence
from the original model, then merges the best trial. This model was abliterated in two iterative
rounds: round 1 removed the dominant refusal direction, then Heretic was re-run on the round-1
model to remove the residual refusal direction that surfaced once the first was gone. Because
ThinkingCap is a thinking model, evaluation ran in non-thinking mode so each judged response is a
real answer rather than an unfinished <think> block; the shipped model restores the original
thinking chat template.
| Round | Refusals | Note |
|---|---|---|
| Baseline | 99/100 | original model |
| Round 1 | 32/100 | dominant refusal direction removed |
| Round 2 | 8/100 | residual direction removed (final) |
mlabonne/harmless_alpaca (good) vs mlabonne/harmful_behaviors (bad).| Scheme | NVFP4, W4A4 |
| Weight rounding | GPTQ (Hessian-based error compensation), MSE observer |
| Weights | FP4 (E2M1), group_size=16, tensor_group, FP8 (E4M3) group scales, shared across fused layers |
| Activations | FP4, dynamic per-group, FP8 (E4M3) scales |
| Quantized | all language-model Linear layers |
| Kept in bf16 | MTP head (model.mtp.*), vision tower (model.visual.*), lm_head |
| Untouched | gated delta-net Conv1d and SSM params (A_log, dt_bias), never Linear |
The recipe ignores re:.*mtp.*, re:.*visual.*, and lm_head. Keeping the MTP head at full
precision is the point: it preserves speculative-decoding acceptance (see above). GPTQ is a
quantization-time cost only; inference speed and format are identical to plain round-to-nearest
NVFP4, but it chooses better 4-bit values.
Calibration: 512 domain-matched samples (long reasoning + general chat + code),
max_seq_len=2048, text-only path through the VL model.
Thinking mode is the default.
temperature=0.6, top_p=0.95, top_k=20temperature=1.0, top_p=0.95, top_k=20temperature=0.7, top_p=0.80, top_k=20{%- set enable_thinking = false %} in the chat template, or
pass extra_body={"chat_template_kwargs": {"enable_thinking": false}}.--speculative-config to use it.--reasoning-parser is not auto-detected; pass it explicitly.chat_template_kwargs.Apache-2.0, following the base model. Intended use and all responsibility for use follow the base model.
vllm serve maci0/ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 \ --served-model-name thinkingcap-27b-abliterated-nvfp4 \ --max-model-len 131072 \ --gpu-memory-utilization 0.90 \ --kv-cache-dtype fp8 \ --reasoning-parser qwen3vllm serve maci0/ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 \ --served-model-name thinkingcap-27b-abliterated-nvfp4 \ --max-model-len 131072 --kv-cache-dtype fp8 --reasoning-parser qwen3 \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'from openai import OpenAIclient = OpenAI(base_url="http://localhost:8000/v1", api_key="x")r = client.chat.completions.create( model="thinkingcap-27b-abliterated-nvfp4", messages=[{"role": "user", "content": "Explain why a heap is better than a sorted array for a priority queue."}],)print(r.choices[0].message.content)