devmaksi

KAT-Coder-V2.5-Dev-NVFP4-freetoken

README

License: apache-2.0

KAT-Coder-V2.5-Dev Highlights

Performance improvement. Through SFT/RL training, KAT-Coder-V2.5-Dev achieves SOTA results in the field of Agentic Coding among models with similar parameter scales.
Optimization of abnormal behaviors. Through RL training, certain abnormal behaviors have been significantly optimized, such as: abnormal tool labels -9pp (9.34% -> 0.28%), single-turn continuous repetition -0.34pp (0.34% -> 0%).

Benchmark performance

For Terminal-Bench 2.1, the bold number is the average across two agent harnesses; the small numbers below are the per-harness scores (Terminus-2 / Claude Code).

Post-training

To provide a systematic overview of our team's work on data and algorithms, we adopt the widely recognized Qwen3.6-35B-A3B as the base model for post-training and build KAT-Coder-V2.5-Dev on top of it. Overall, KAT-Coder-V2.5-Dev largely follows the post-training recipe of KAT-V2.5, with most settings—including data construction, training pipeline, and optimization strategy—remaining unchanged. The full pipeline consists of two stages: supervised fine-tuning (SFT) and reinforcement learning (RL). We first fine-tune Qwen3.6-35B-A3B on a dataset of 127K examples and then perform RL training on the resulting SFT model.

During the RL stage, we retain the training infrastructure and key technical designs validated in KAT-V2.5, including the following four components:

Token-in-Token-out (TITO) consistency. We use TITO to ensure that the token sequences in the rollout and training stages are strictly identical, preventing training discrepancies caused by differences in chat templates, serialization, or tokenizer behavior.
Truncated Importance Sampling (TIS). To mitigate policy staleness and off-policy issues introduced by asynchronous rollouts, we apply TIS to truncate importance-sampling weights, reducing the variance and instability caused by excessively large weights.
Reliable sandboxes and verifiers. We systematically inspect and validate the stability and correctness of the sandboxes and verifiers. This helps prevent infrastructure failures—such as execution timeouts, environment errors, or verifier misjudgments—from being incorrectly treated as model failures and contaminating the reward signal.
Hierarchical rewards based on harness execution feedback. We construct hierarchical rewards from fine-grained execution feedback provided by the harness. This allows the model to optimize toward the final task objective while also receiving credit for meaningful progress in unsuccessful trajectories, thereby increasing the training value of failed attempts and providing denser reward signals.

However, Qwen3.6 exhibits trajectory patterns that differ from those observed in KAT-V2.5, requiring additional reward adaptations tailored to its behavior. In our initial experiments, a simple binary 0–1 reward caused model collapse as early as the second epoch. An analysis of the training trajectories revealed that, as training progressed, the model increasingly tended to issue a large number of parallel tool calls within a single turn—occasionally exceeding 70 calls. This behavior caused the context length to grow rapidly, generated a substantial number of invalid trajectories and execution errors, and ultimately destabilized RL training.

To address this issue, we augmented the original hierarchical reward with several Qwen3.6-specific penalties targeting (including but not limited to the following):

Excessive parallel tool calls within a single turn;
Failed tool calls;
Empty tool-call blocks; and
Large amounts of repeated content.

These targeted reward adjustments effectively suppressed pathological tool-use and repetitive-generation behaviors, enabling stable RL training for 10 epochs. Our experiments validate the effectiveness and feasibility of both the overall training pipeline and the Qwen3.6-specific reward design.

About this NVFP4 quantization

This directory is a locally quantized copy of KAT-Coder-V2.5-Dev (bf16). The original checkpoint in ../KAT-Coder-V2.5-Dev is untouched.

What changed vs the unquantized model

Table with columns: Component, Was (bf16), Now, Why
Component	Was (bf16)	Now	Why
Routed MoE experts (gate/up/down, 256 experts x 40 layers)	bf16, 65 GB total	NVFP4: packed e2m1 codes (2 elems/byte) + e4m3 per-16-element block scales + per-tensor global scales (ModelOpt convention: `w = fp4 * scale * (1/weight_global_scale)`)	The experts are ~90% of all weights. FP4 cuts them ~4x; on Blackwell (RTX 50-series) FP4 runs on tensor cores. The MoE experts are read by FreeToken's offload cache from the packed NVFP4 banks without dequantization
Attention and GDN projections (q/k/v/o, in_proj_qkv, in_proj_z, out_proj)	bf16	Block FP8, 128x128 blocks: e4m3 weights + bf16 scale_inv	FP8 keeps attention quality near-lossless (cosine 0.9997 vs bf16 on test tensors) and these layers are latency-critical
Router, shared expert, in_proj_b/a, norms, embeddings, lm_head	bf16

Overall: 65 GB -> 21 GB (6 shards). Weight layout follows the compressed-tensors (llm-compressor) NVFP4 dialect that FreeToken reads for the Qwen3.5-MoE architecture.

Quality

Round-trip through FreeToken's own NVFP4 dequant kernel: cosine 0.994, ~10% mean relative error - typical for RTN FP4 without calibration.
Verified live: server starts, /v1/chat/completions produces coherent code (reasoning + answer), long-context prefill and decode work at 128k and 200k.

Measured throughput (RTX 5080 16GB, KV cache 210k tokens)

Table with columns: Context, Prefill, Decode
Context	Prefill	Decode
128k tokens	~28 s (~4,550 tps)	~84 tps
200k tokens	~49 s (~4,000 tps)	~78 tps

Reproducing

quantize_kat_nvfp4.py in the FreeToken repository root reproduces this checkpoint from the bf16 original.

Quickstart

For streamlined integration, we recommend using KAT-Coder-V2.5-Dev via APIs. Below is a guide to use KAT-Coder-V2.5-Dev via OpenAI-compatible API.

Serving the NVFP4 checkpoint with FreeToken

FreeToken reads this NVFP4 checkpoint directly - no conversion step needed. FreeToken serves OpenAI /v1/*, Anthropic /v1/messages and Responses APIs, with chunked prefill, paged KV / radix cache and MoE expert offload.

Note: serve with --text-model-only: this checkpoint carries no vision weights, and the Qwen3_5MoeForConditionalGeneration architecture makes FreeToken build (and expect) a vision tower unless disabled.

From the FreeToken repository root:

shell
./run-kat-nvfp4.sh

This starts the API server on http://127.0.0.1:1919 with sensible defaults for this checkpoint. Equivalent manual command:

shell
ft serve \
    --model /path/to/KAT-Coder-V2.5-Dev-NVFP4 \
    --text-model-only \
    --port 1919

Tool use and reasoning are auto-detected from the checkpoint (Qwen tool-call parser, reasoning parser splits chain-of-thought into reasoning_content).

Long context: the server sizes the KV cache from free VRAM automatically. To raise it explicitly (e.g. for 200k+ tokens of headroom on a 16 GB GPU):

shell
ft serve --model /path/to/KAT-Coder-V2.5-Dev-NVFP4 --text-model-only --port 1919
# then, live, without a restart:
ft ctl cache --kv 210000 --moe 2000 --wait 300

Chat in the terminal:

shell
ft shell --model /path/to/KAT-Coder-V2.5-Dev-NVFP4

Hook up a coding agent (Claude Code, Codex, OpenCode, ...):

shell
ft launch opencode

See the FreeToken CLI reference for all options.

Note: this NVFP4 checkpoint is in FreeToken's compressed-tensors NVFP4 dialect. Other frameworks (transformers, KTransformers) do not read this format; use the original bf16 checkpoint for them.

Using KAT-Coder-V2.5-Dev via the Chat Completions API

The chat completions API is accessible via standard HTTP requests or OpenAI SDKs. Here, we show examples using the OpenAI Python SDK.

Before starting, make sure it is installed and the API key and the API base URL is configured, e.g.:

shell
pip install -U openai

# Set the following accordingly
export OPENAI_BASE_URL="http://localhost:8000/v1"
export OPENAI_API_KEY="EMPTY"

Text-Only Input

python
from openai import OpenAI
# Configured by environment variables
client = OpenAI()

messages = [
    {"role": "user", "content": "Type \"I love KAT-Coder-V2.5-Dev\" backwards"},
]

chat_response = client.chat.completions.create(
    model="Kwaipilot/KAT-Coder-V2.5-Dev",
    messages=messages,
    max_tokens=81920,
    temperature=1.0,
    top_p=0.95,
    presence_penalty=1.5,
    extra_body={
        "top_k": 20,
    },
)
print("Chat response:", chat_response)

Instruct (or Non-Thinking) Mode

KAT-Coder-V2.5-Dev will think by default before response. You can obtain direct response from the model without thinking by configuring the API parameters. For example,

python
from openai import OpenAI
# Configured by environment variables
client = OpenAI()

messages = [
    {"role": "user", "content": "Write a Python function that returns the n-th Fibonacci number."},
]

chat_response = client.chat.completions.create(
    model="Kwaipilot/KAT-Coder-V2.5-Dev",
    messages=messages,
    max_tokens=32768,
    temperature=0.7,
    top_p=0.8,
    presence_penalty=1.5,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": False},
    },
)
print("Chat response:", chat_response)

Preserve Thinking

By default, only the thinking blocks generated in handling the latest user message is retained, resulting in a pattern commonly as interleaved thinking. KAT-Coder-V2.5-Dev has been additionally trained to preserve and leverage thinking traces from historical messages. You can enable this behavior by setting the preserve_thinking option:

python
from openai import OpenAI
# Configured by environment variables
client = OpenAI()

messages = [...]

chat_response = client.chat.completions.create(
    model="Kwaipilot/KAT-Coder-V2.5-Dev",
    messages=messages,
    max_tokens=32768,
    temperature=0.7,
    top_p=0.8,
    presence_penalty=1.5,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"preserve_thinking": True},
    },
)
print("Chat response:", chat_response)

This capability is particularly beneficial for agent scenarios, where maintaining full reasoning context can enhance decision consistency and, in many cases, reduce overall token consumption by minimizing redundant reasoning. Additionally, it can improve KV cache utilization, optimizing inference efficiency in both thinking and non-thinking modes.

Processing Ultra-Long Texts

KAT-Coder-V2.5-Dev natively supports context lengths of up to 262,144 tokens. For long-horizon tasks where the total length (including both input and output) exceeds this limit, we recommend using RoPE scaling techniques to handle long texts effectively, e.g., YaRN.

YaRN is configured via the model configuration file: in config.json, change the rope_parameters fields in text_config to:

json
{
    "mrope_interleaved": true,
    "mrope_section": [
        11,
        11,
        10
    ],
    "rope_type": "yarn",
    "rope_theta": 10000000,
    "partial_rotary_factor": 0.25,
    "factor": 4.0,
    "original_max_position_embeddings": 262144
}

FreeToken reads max_position_embeddings from the checkpoint, so after the config edit the server picks the extended length up automatically:

shell
./run-kat-nvfp4.sh

Citation

If you find our work helpful, feel free to give us a cite.

bibtex
@misc{katcoder_v25_2026,
  title={{KAT-Coder-V2.5 Technical Report}},
  author={{KwaiKAT Team}},
  year={2026},
  month={July},
  eprint={2607.05471},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/pdf/2607.05471}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider