beyoru

KAT-Coder-V2.5-Dev-VL

README

License: apache-2.0

Vision capability — READ THIS

The vision tower was never co-trained with this checkpoint's language model, which was heavily RL-tuned for agentic coding. The vision→LM projection therefore carries coarse features reliably and fine features unreliably. Measured on a served endpoint, greedy decoding:

Table with columns: probe, score, notes
probe	score	notes
Dominant colour (4 solid-colour images)	4 / 4	red, blue, green, yellow — all correct
Shape (circle / square / triangle)	3 / 3	all correct
Scene description (real icon)	plausible	correctly identified colour and rough form
Text/number reading (OCR)	0 / 4	`CAT`→`CCT`, `42`→`48`, `HELLO`→`Hiro`, `7492`→`ZOE`

The OCR failures are all near-misses onto visually similar glyphs, which is the signature of a tower that resolves stroke-level shape but not glyph identity. In practice:

✅ Reasonable for: "is there a chart here", colour/layout/shape questions, coarse scene gist, "does this screenshot show an error dialog or a code editor".
❌ Not usable for: reading code from screenshots, OCR, reading UI labels, dense-document understanding, chart-value extraction, anything where a wrong character changes the answer.

If your workload is text-only, prefer upstream Kwaipilot/KAT-Coder-V2.5-Dev — this build offers no advantage there (the LM weights are identical) and costs 0.9 GB extra VRAM.

Ready-made configs

Table with columns: file, what, status
file	what	status
`docker-compose.vllm.yaml`	vLLM, 1 GPU, vision enabled	verified
`docker-compose.sglang-flashqla.yaml`	SGLang + FlashQLA GDN prefill	untested

FlashQLA applies here because 30 of the 40 layers are linear_attention (Gated DeltaNet) — exactly the layers its TileLang kernel accelerates during prefill. It is opt-in via a single flag (--linear-attn-prefill-backend flashqla); the default path is byte-identical to stock SGLang.

Windowed-MTP does not apply to this model. mtp_num_hidden_layers = 0, so the checkpoint carries no MTP/NEXTN head to self-speculate with. That technique needs Qwen/Qwen3.6-35B-A3B (mtp = 1).

Serving

vLLM (recommended, verified)

Verified working configuration:

Table

Image	`vllm/vllm-openai:v0.26.0`
GPU	1 × H200 (143 GB), tensor-parallel 1
Load time	~2 min cold (14 shards, 65.4 GiB)
Attention	FlashAttention 3 (LM), FLASH_ATTN (ViT), FlashInfer GDN for linear attention
MoE backend	TRITON (auto-selected)

bash
docker run -d --name kat-vl \
    --gpus '"device=0"' --ipc=host \
    -v /path/to/models:/models \
    -p 8000:8000 \
    vllm/vllm-openai:v0.26.0 \
    --model /models/KAT-Coder-V2.5-Dev-VL \
    --served-model-name kat-vl \
    --max-model-len 32768 \
    --max-num-batched-tokens 8192 \
    --gpu-memory-utilization 0.90 \
    --trust-remote-code

--max-num-batched-tokens 8192 is required. This is a hybrid attention + Gated-DeltaNet architecture. With prefix caching on (the default in recent vLLM), vLLM forces mamba_cache_mode='align' and raises the attention block size to 2096 so that "attention page size >= mamba page size". The default max_num_batched_tokens is 2048, so startup dies with
markdown
AssertionError: In Mamba cache align mode, block_size (2096) must be <= max_num_batched_tokens (2048)
. Any value >= 2096 works.

Or without Docker (vllm>=0.26.0):

bash
uv pip install vllm --torch-backend=auto
vllm serve beyoru/KAT-Coder-V2.5-Dev-VL \
    --served-model-name kat-vl \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90 \
    --trust-remote-code

Do NOT pass --language-model-only. That flag is required for the upstream text-only release; passing it here disables the vision tower and you get the upstream model back.

Memory. bf16 weights are 70.2 GB, so a single 80 GB card cannot hold them plus a usable KV cache. Use one 141/143 GB card, or --tensor-parallel-size 2 on 2 × 80 GB. Raise --max-model-len only after checking free memory — the model's native 262 144 context needs a large KV allocation.

Long context. For contexts beyond the native window, apply YaRN exactly as documented for Qwen/Qwen3.6-35B-A3B; the RoPE parameters here are unchanged from upstream. Note that all major frameworks implement static YaRN, so enabling it can hurt short-prompt quality — turn it on only when you actually need long contexts.

SGLang

sglang>=0.5.10. Because this build does contain the vision tower, do not pass the text/language-model-only flag that the upstream release requires.

Transformers

Works out of the box — verified on transformers==5.10.1.

python
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
    "beyoru/KAT-Coder-V2.5-Dev-VL", dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained("beyoru/KAT-Coder-V2.5-Dev-VL")

Note: this checkpoint stores MoE experts in the per-expert (split) layout (...mlp.experts.{0..255}.{gate,up,down}_proj.weight), inherited unchanged from upstream, while transformers 5.x uses a fused layout internally. The conversion is automatic at load time — no manual step is needed, and no weights are lost.

Usage

The API is OpenAI-compatible. This is a thinking model: it emits a reasoning block terminated by </think> before the answer, so give it a generous max_tokens (≥256) or you will only ever receive truncated reasoning. Strip everything up to and including </think> to get the final answer, or disable thinking with chat_template_kwargs: {"enable_thinking": false}.

python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

# text
r = client.chat.completions.create(
    model="kat-vl",
    messages=[{"role": "user", "content": "Write a Python function to reverse a linked list."}],
    max_tokens=512, temperature=0.0,
)
print(r.choices[0].message.content.split("</think>")[-1].strip())

# image + text
r = client.chat.completions.create(
    model="kat-vl",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}},
        {"type": "text", "text": "Describe this image in one sentence."},
    ]}],
    max_tokens=512, temperature=0.0,
)
print(r.choices[0].message.content.split("</think>")[-1].strip())

Sampling parameters follow upstream: temperature=1.0, top_p=0.95, top_k=20 for open-ended generation; temperature=0.0 for deterministic coding tasks.

What this build is and is not

Is: the upstream language model, byte-for-byte, plus a working vision tower and a patched weight index so the declared multimodal architecture loads.
Is not: a multimodally trained model. The vision path was not fine-tuned against this LM. Treat its visual outputs as coarse hints, never as ground truth.
No distillation, no re-training, no quantization, no merging of language-model weights.

Acknowledgements

Kwaipilot/KAT-Coder-V2.5-Dev — the base model and all language-model weights (Apache-2.0).
Qwen/Qwen3.6-35B-A3B — the architecture this model derives from (Apache-2.0).

Both are Apache-2.0; this build is released under the same licence.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

beyoru

Model Tree

Base

Kwaipilot/KAT-Coder-V2.5-Dev

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality