Credits
Huge thanks to BottleCap AI for releasing ThinkingCap. Their checkpoint is the model here; this repo is only a community quantization of it.
Also, personally: BottleCap are local friends/neighbours — their offices are about 200 meters from me, and I have known them for years — so this one is especially fun to package properly. Please give the original model and team credit:
What is ThinkingCap?
From the original BottleCap release: ThinkingCap keeps the capability and style of Qwen3.6-27B while using substantially fewer thinking tokens on average. See the original model card for the full evaluation table, methodology, citation, and intended sampling settings.
This quant keeps the base/default sampling recipe:
temperature = 1.0
top_p = 0.95
top_k = 20
min_p = 0.0
Quantization recipe
Table with columns: Field, Value| Field | Value |
|---|
| Source model | bottlecapai/ThinkingCap-Qwen3.6-27B |
| Quantizer | Intel AutoRound |
| AutoRound version | 0.14.0 |
| Scheme | W4A16 / INT4 weight-only |
| Bits | 4 |
| Group size | 128 |
| Symmetric | true |
| Packing format | auto_round:auto_gptq |
| Quantized block | |
Notes:
- This is the group-size 128 release. It prioritizes speed, smaller resident VRAM, and long-context KV headroom.
- Smaller group sizes such as GS32/GS16 may improve quality but increase metadata size and reduce KV-cache room. A later v2 may explore that direction.
- MTP/speculative-decoding tensors were restored from the BF16 source auxiliary tensors instead of leaving them packed/quantized.
Recommended long-context / agent serving config
This is the more tuned single-GPU configuration used during local testing:
vllm serve "$MODEL_ID" \
--dtype half \
--max-model-len 262144 \
--gpu-memory-utilization 0.92 \
--kv-cache-dtype turboquant_4bit_nc \
--max-num-seqs 16 \
--max-num-batched-tokens 4096 \
--enable-chunked-prefill \
--max-long-partial-prefills 1 \
--long-prefill-token-threshold 4096 \
--compilation-config '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,2,3,4],"max_cudagraph_capture_size":4}' \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--default-chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}' \
--port 8888 \
--host 0.0.0.0 \
--trust-remote-code \
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
Notes:
turboquant_4bit_nc is a vLLM KV-cache quantization mode, not the model weight quantization. It is useful for fitting long context.--max-num-seqs 16 allows many shorter concurrent requests; it does not mean sixteen full 262k requests fit simultaneously.--compilation-config ... PIECEWISE ... [1,2,3,4] keeps CUDA graphs enabled but caps graph capture sizes to save VRAM.- vLLM 0.24 rejected non-default concurrent partial prefill in this environment, so the command intentionally does not pass
--max-num-partial-prefills 4.
BenchLocal snapshots
Local BenchLocal screenshots; preliminary smoke/quality signals, not a complete public evaluation suite. Run target: vllm 5090 on the local W4G128 checkpoint.
Hermes Agent benchmark
The main benchmark for this release: a Hermes-style agent suite covering memory, scheduling, delegation/recovery, procedural skills, and workspace orchestration.
Table with columns: Overall, memory_recall, scheduling_delivery, delegation_recovery_boundaries, skills_procedural_memory, workspace_orchestration| Overall | memory_recall | scheduling_delivery | delegation_recovery_boundaries | skills_procedural_memory | workspace_orchestration |
|---|
| 81 | 93 | 75 | 76 | 80 | 83 |
Table with columns: Outcome, Cases| Outcome | Cases |
|---|
| Pass, 13/20 | HA-01, HA-02, HA-04, HA-05, HA-06, HA-08, HA-09, HA-10, HA-11, HA-13, HA-15, HA-18, HA-20 |
| Fail, 5/20 | HA-07, HA-12, HA-14, HA-16, HA-17 |
| Provider warning, 2/20 | HA-03, HA-19 |
Other local BenchLocal runs
Table with columns: BenchLocal set, Overall| BenchLocal set | Overall |
|---|
| TC-15 / tool-call-style set | 90 |
| instructfollow-15 | 89 |
| bugfind-15 | 73 |
| cli-40 | 65 |
Local inference-benchmarker tests against vLLM used ~256 input tokens and ~64 output tokens with thinking disabled for content-throughput measurement. The tuned server reached approximately:
Table with columns: Arrival rate, Actual QPS, Output tok/s, Avg TTFT, Avg ITL, Success| Arrival rate | Actual QPS | Output tok/s | Avg TTFT | Avg ITL | Success |
|---|
| 1 req/s | ~1.01 | ~65 tok/s | ~79 ms | ~9 ms | 29/29 |
| 3 req/s | ~2.96 | ~187 tok/s | ~110 ms | ~11 ms | 87/87 |
| 6 req/s | ~5.77 |
These are local measurements on one machine and are not a formal quality benchmark. They are included to show that the quant is quick and usable under concurrent short-request traffic.
Compatibility
Known-good path:
- vLLM
0.24.0
- CUDA 13.x on RTX 5090
--kv-cache-dtype turboquant_4bit_nc- MTP speculative decoding with
num_speculative_tokens=1
The model should also be usable through Transformers/GPTQModel-compatible AutoRound loaders, but the most exercised runtime for this artifact is vLLM.
Limitations
- This is an INT4 quantization; it is not expected to exactly match BF16 quality.
- Group size 128 is chosen for practicality. Future v2 variants may trade more disk/VRAM for quality.
- The included performance numbers are local serving benchmarks, not broad model evaluations.
- For long-context serving, KV-cache settings matter as much as model size. If you disable KV quantization, reduce context/concurrency accordingly.
Citation
If you use this quant, please cite and credit the original BottleCap model. Original citation from their model card:
@misc{,
title = {bottlecapai/ThinkingCap-Qwen3.6-27B},
author = {Lasocki, Karol and Osusky, Adam and Lindauer, Jan and Jirkovsky, Adam and Mihal, Filip and Platek, Ondrej and Herel, David and Ihnatchenko, Luka and Bartek, Vojtech and Jirak, Jiri and Mikolov, Tomas},
year = {2026},
}