Ttimms

KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16

README

License: apache-2.0

Highlights

Table with columns: Result, Detail
Result	Detail
12.45 GiB	REAP 50% expert pruning + NVFP4A16 (weight-only, data-free), vision tower stripped
149.5 tok/s median, n=5	Benchmark config: 512 in / 256 out, batch 1, CUDA graphs (FULL_AND_PIECEWISE), 14,672-token context ceiling
89.0% / 90.5%	HumanEval+ [83.3, 92.9] / MBPP+ [87.1, 93.0], greedy, instruct framing
52.0% (26/50)	SWE-bench Verified, `mini-swe-agent` bash-only, 49K context — see caveats below
28.9 s load	CUDA graphs enabled, no CPU offload

Why 50 percent

Forced by arithmetic on a 16 GB card, not a tuning choice:

Table with columns: variant, size, fits 16 GB
variant	size	fits 16 GB
bf16 base	69.3 GB	no
NVFP4, unpruned	21.9 GB	no
REAP 25% + NVFP4	~16–17 GB	no — not once KV cache is counted
REAP 50% + NVFP4	12.45 GiB	yes

Supporting evidence: Half the Experts, All the Code pruned Qwen3.6-35B-A3B, this model's own base, at 50% with no statistically detectable loss on its primary code benchmark.

SWE-bench Verified — read before citing the 52.0% figure alone

Table with columns: metric, value
metric	value
resolved	26/50 = 52.0%
resolved of completed (valid patch produced and tested)	26/32 = 81.25%
ContextWindowExceeded	17 (49K ceiling)
garbage/invalid patch (patch failed to apply)	1 of 33 generated
ran, tests failed (genuinely unresolved)	6

Prior measurement at this checkpoint's original 32K-token ceiling,

markdown

step_limit 40

: 20/50 = 40.0% (18 ContextWindowExceeded, 9 LimitsExceeded — ran out of agent turns before finishing, a different failure mode than the context ceiling — 20/22 = 90.9% resolved-of-completed). Raising both the context ceiling to 49K and the step limit to 65 (kat_overrides_sota.yaml) moved the headline number from 40.0% to 52.0%, but not primarily by reducing context-window failures — that rate barely moved (17/50 = 34% vs. 18/50 = 36%). The real lever was the step limit: LimitsExceeded failures went from 9 to 0, so 10 more instances (22→32) reached a real completion attempt instead of running out of turns first. Those newly-reachable instances resolve at a lower rate than the ones that were already completing (81.25% resolved-of-completed at 49K vs. 90.9% at 32K — consistent with them being the harder, longer problems that need the extra turns), but enough resolved anyway that the net resolved count still rose (20→26). The scaffold's context ceiling is still the safe limit this card's VRAM budget supports, not a property of the model — Devstral Small averages 86.9 LM calls/instance under the same scaffold, and instances needing more than 49K tokens of context or more than 65 agent turns still fail to submit at all. This is disclosed as a real result, not an excuse: 52.0% is the correct number to cite; the breakdown above is the correct context for interpreting it.

Config note (2026-08-23): kat_overrides_sota.yaml — the config behind the 52.0% figure above — is kept byte-identical to the run that produced it; a fresh clone reproduces this exact result by default. A candidate change (presence_penalty/top_k, completing this base model's own documented sampling recommendation) was tested full-pilot on the same 50 instances and regressed the score to 24/50 = 48.0% — more instances ran out of the turn budget exploring alternatives (LimitsExceeded 0→8) than were saved from the context ceiling (ContextWindowExceeded 17→14). Not promoted; kat_overrides_sota.yaml is unchanged. Kept in the repo as a documented negative result (kat_overrides_sota_presence_penalty.yaml), not deleted.

Prior art and scope of claims

Verified against the Hugging Face Hub on 2026-08-17:

REAP combined with NVFP4 on qwen3_5_moe already exists (rene98c/Qwen3.5-397B-A17B-REAP-28-NVFP4, March 2026, 23.1K downloads).
REAP on this specific model exists as GGUF (gbuzhf/KAT-Coder-V2.5-Dev-REAP-205E-MTP-GGUF).

What is distinct, and all that is claimed: a vLLM-servable KAT-Coder that is genuinely usable in 16 GB, with published SWE-bench Verified, HumanEval+, and MBPP+ numbers and their confidence intervals — none of which the prior art above publishes.

Quantization and pruning details

Table with columns: Field, Value
Field	Value
Base model	`Kwaipilot/KAT-Coder-V2.5-Dev`
Pruning	REAP, expert-level, 50% compression ratio, seed 42
Pruning calibration	`theblackcat102/evol-codealpaca-v1`, 64 batches/category, 2048 max length
Router renormalization	Fixed (was silently disabled by the upstream REAP adapter for this architecture; committed for upstreaming)
Quantization method	compressed-tensors / llm-compressor, `QuantizationModifier` (PTQ)
Quantization scheme	NVFP4A16 — weight-only, data-free, 82 s

Usage

Requires vLLM with SM120 support (CUDA graphs are correct on this card for this model, despite past reports of SM120 CUDA-graph issues on other architectures) and native tool calling for agentic use:

bash
vllm serve Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 \
  --served-model-name kat-16gb \
  --max-model-len 49152 --max-num-seqs 2 \
  --gpu-memory-utilization 0.92 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --enable-prefix-caching --max-num-batched-tokens 4096 \
  --compilation-config '{"cudagraph_capture_sizes":[1,2],"cudagraph_mode":"PIECEWISE"}' \
  --language-model-only

--language-model-only is required: the model declares a vision tower it has no trained weights for, and without this flag vLLM profiles a 16K-token image budget through it. --enable-prefix-caching is the single largest agentic lever measured on this model — 45x on replayed history (0.21 s vs 30.74 s for a 13,130-token history). Never set --max-model-len near a measured ceiling: available KV cache swings 0.49–1.41 GiB with host desktop VRAM use, and higher values fail intermittently rather than at startup.

Sampling: temperature=1.0, top_p=0.95 is what the published SWE-bench result used and is the current recommendation. Kwaipilot/KAT-Coder-V2.5-Dev's own model card documents two more params alongside these, presence_penalty=1.5, top_k=20, for Thinking mode. Tested on this checkpoint: on a single instance it suppressed a genuine repetition-loop failure, but a full-pilot test on 50 instances found the net effect regresses the agentic score (see the SWE-bench config note above) — not recommended for agentic use on this checkpoint despite matching the base model's own documented config.

Evaluation

HumanEval+ and MBPP+ via lm-eval-harness / EvalPlus, greedy decoding, instruct framing, Wilson confidence intervals:

Table with columns: benchmark, score, 95% CI, n
benchmark	score	95% CI	n
HumanEval+	89.0%	[83.3, 92.9]	164
MBPP+	90.5%	[87.1, 93.0]	378

KAT-Coder-V2.5-Dev publishes no HumanEval/MBPP/EvalPlus numbers, so there is no published upstream figure to compare these against.

Both figures were re-measured on the released checkpoint itself and reproduced inside their intervals: HumanEval+ 90.9% [85.5, 94.4] and MBPP+ 89.9% [86.5, 92.6], same problem counts, same greedy decoding. The table reports the original measurement. The two differences run in opposite directions (+1.9 pp and -0.6 pp), which is greedy-decoding nondeterminism under vLLM's batching rather than a difference in weights. Reproduce with bash scripts/eval/eval_suite.sh.

SWE-bench Verified via the official swebench.harness.run_evaluation harness against mini-swe-agent bash-only rollouts (scaffold: SWE-bench/experiments v1.17.2 configuration) — see the dedicated section above for the full breakdown and required caveats.

Known limitations

49K context window (raised from an earlier 32K ceiling) is a hardware-forced limit, not a design choice — this card's KV-cache budget cannot safely support more. SWE-bench results must still be read as context-limited: 17 of 50 instances in the reported run failed purely from exceeding this ceiling, not from the model failing the task.
No pruning-ablation baseline measured. The unpruned model is 69.3 GB bf16 and does not fit this hardware; the accuracy cost of pruning itself (independent of quantization) is not isolated here.
SWE-bench Verified no longer accepts leaderboard submissions outside academia — these numbers are self-reported and independently reproducible from the released evaluation scripts, not a leaderboard entry.

W4A4: an alternative quantization strategy

A follow-up build, Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4-W4A4, re-quantizes this same pruned checkpoint to NVFP4 (W4A4 — weights and activations to 4 bits) to reach native FP4 tensor-core kernels instead of this model's Marlin dequant-to-bf16 fallback. Measured properly (5 interleaved runs per arm, median + range, under both isolated eager mode and the same PIECEWISE CUDA-graph configuration this model actually serves with): W4A4 decodes at 119.2 tok/s vs this checkpoint's 142.5 tok/s (0.84x) with a mixed accuracy picture (HumanEval 92.07% vs 95.7%, HumanEval+ 89.02% vs 90.9%, MBPP+ 91.01% vs 89.9% — that last one favors W4A4). This checkpoint is faster on this hardware and is what we default to; the W4A4 build is a legitimate alternative if the native FP4 execution path matters more for your use case. Only single-stream (batch=1) decode was measured. Full writeup: ROADMAP.md's RESULT section in the release repo.

GPTQ: a tested rounding variant, not a recommended alternative

A third build, Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16-GPTQ, re-quantizes the same pruned checkpoint to the identical NVFP4A16 scheme using GPTQ rounding instead of this release's round-to-nearest (RTN) — same size (12.4512 GiB, exact match), same kernel, only the quantization algorithm differs. Unlike W4A4 above, this is not published as an alternative worth choosing — a paired McNemar test found no statistically significant difference from this release on either HumanEval+ (p=0.68) or MBPP+ (p=0.81), and there's no distinct execution-path reason to prefer it the way there is for W4A4. It's published for completeness and independent verification of that null result, not as a recommendation. Full writeup on that card and in ROADMAP.md's RESULT section in the release repo.

License

Apache 2.0, inherited from the base model Kwaipilot/KAT-Coder-V2.5-Dev and matching the reap and llm-compressor toolchains used to build this checkpoint.

Citation

This checkpoint is derived from Kwaipilot/KAT-Coder-V2.5-Dev. If you use it, please cite the upstream technical report:

bibtex
@misc{katcoder_v25_2026,
  title={{KAT-Coder-V2.5 Technical Report}},
  author={{KwaiKAT Team}},
  year={2026},
  month={July},
  eprint={2607.05471},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/pdf/2607.05471}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Ttimms

Model Tree

Base

Kwaipilot/KAT-Coder-V2.5-Dev

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality