Architecture
Qwen3_5ForConditionalGeneration (model_type qwen3_5), dense 27.8 B:
- Hybrid attention — 48 Gated-DeltaNet (linear) + 16 full-attention layers (64 total,
full_attention_interval=4), hidden 5120, 262 K native context.
- Vision — Qwen3-VL ViT (1152→5120), kept bf16; serve text-only with
--limit-mm-per-prompt.
- Native MTP (
mtp_num_hidden_layers=1), kept bf16 → drives vLLM speculative decoding.
- Thinking-by-default reasoning model (
<think>…</think>, use --reasoning-parser qwen3) — but a token-efficient one.
Quantization recipe
QuantizationModifier(targets="Linear", scheme="NVFP4", # W4A4, group_size 16
ignore=["lm_head", "re:.*visual.*", "re:.*conv1d.*", "re:.*mtp.*"])
- Vision tower, DeltaNet causal
conv1d, lm_head, and the entire MTP head stay bf16; everything else is NVFP4 W4A4. 32 calibration samples (neuralmagic/calibration), seq 8192, pure-CPU load (sequential-pipeline onload).
- ThinkingCap ships the MTP head as a separate
model-base-aux.safetensors (15 bf16 tensors). Those are grafted into the NVFP4 output (model-mtp-bf16.safetensors) and spliced into the safetensors index.
- Note for re-bakers: the grafted MTP modules must also be added to
quantization_config.ignore, otherwise vLLM matches mtp.*_proj against targets=["Linear"], expects NVFP4 scales that do not exist, and loads the Qwen3_5MTP draft as garbage → 0 % spec-decode acceptance. This bake adds them automatically.
Serving (vLLM ≥ 0.21)
vllm serve sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4 \
--tensor-parallel-size 4 --max-model-len 131072 \
--max-num-seqs 16 --gpu-memory-utilization 0.90 --kv-cache-dtype fp8 \
--reasoning-parser qwen3 --limit-mm-per-prompt '{"image":0,"video":0}' \
--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
On NVLink-less boxes add NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce (and NCCL_CUMEM_ENABLE=0 if TP=8 CUDA-graph capture hangs). Drop --speculative-config for plain decode. The hybrid model's KV is light (only the 16 full-attention layers cache), so full 128 K context fits even at TP=2.
- Reasoning model → set
max_tokens ≥ 4096 (prefer 8192+). Even though ThinkingCap thinks less, at a tiny budget it can still spend it all inside <think> and return empty content.
- Do not produce a W4A16 / NVFP4A16 variant — it fails to serve on vLLM (
gptq_marlin_repack: size_n=24 not divisible by tile_n_size=64; the 24 attention-heads / DeltaNet odd dims violate Marlin's tile constraint). W4A4 avoids Marlin (NVFP4 cutlass/FlashInfer path).
- Sampling: the base recommends
temperature=1.0, top_p=0.95, top_k=20.
License & attribution
Apache-2.0, inherited from the base models. Token-efficiency fine-tune by BottleCap AI; base Qwen3.6-27B by the Qwen Team. NVFP4 quantization by sakamakismile (Lna-Lab), reusing the validated qwen3_5 dense+MTP recipe shared with sakamakismile/Qwen3.6-27B-MTP-pi-tune-NVFP4.