beyoru

KAT-Coder-V2.5-Dev-VL-Flash

README

License: apache-2.0

Loading status on Blackwell

Confirmed loading and initialising on sm121 with vLLM (nightly 0.1.dev18498):

markdown
Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend
Loading weights took 114.37 s -> 20.61 GiB
FLASH_ATTN for vit attention / MMEncoderAttention     <- vision tower initialises
torch.compile 36.83 s · CUDA graphs PIECEWISE=51 FULL=35 · KV cache 83.33 GiB

Weights, NVFP4 kernel selection, the MoE backend, the vision tower, compilation and CUDA-graph capture all come up cleanly. Generation quality on FP4 tensor cores has still not been benchmarked — no accuracy numbers are claimed here. Reports in the Community tab are welcome.

Precision layout

Mixed precision — not everything is FP4. Weight-only 4-bit (W4A16) does not touch FP4 tensor cores, so the parts that dominate decode are W4A4, while the parts that hurt accuracy most stay higher.

Table with columns: Component, Precision
Component	Precision
MoE experts, all 40 layers (`mlp.experts.*.{gate,up,down}_proj`)	NVFP4 W4A4 (group 16, FP8-E4M3 scales)
Shared expert (`shared_expert.{gate,up,down}_proj`)	NVFP4 W4A4
Attention (`self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`)	FP8 W8A8
`lm_head`	FP8 W8A8
Router (`mlp.gate`, `shared_expert_gate`), `linear_attn.{in_proj_a,in_proj_b,norm}`	BF16
Vision tower (all `model.visual.*`)	BF16
KV cache	FP8 (calibrated, per-tensor static)

Pushing every expert layer to W4A4 is the "Fast" trade: maximum FP4 tensor-core coverage, at some accuracy cost versus keeping the last layers in FP8.

Table with columns: bf16 build, this build
	bf16 build	this build
Size on disk	70.2 GB	see Files and versions
Minimum GPU	1 × 141 GB, or 2 × 80 GB	fits comfortably on a single Blackwell card

Ready-made config

docker-compose.vllm.yaml — vLLM on a single Blackwell GPU, vision enabled, MoE backend left to auto-selection. Untested on real FP4 hardware (see the warning above).

SGLang users: FlashQLA's GDN prefill kernel applies to this architecture (30 of 40 layers are linear_attention) and is opt-in via --linear-attn-prefill-backend flashqla. Windowed-MTP does not apply — mtp_num_hidden_layers = 0, so there is no MTP/NEXTN head to self-speculate with.

Serving

vLLM

bash
uv pip install vllm --torch-backend=auto      # vllm >= 0.26.0

vllm serve beyoru/KAT-Coder-V2.5-Dev-VL-Flash \
    --served-model-name kat-vl-flash \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90 \
    --trust-remote-code

Let vLLM pick the MoE backend. Do not pin one. On Blackwell the CUTLASS / FlashInfer-TRTLLM / CuTe-DSL NVFP4 paths are the fast ones; forcing Marlin gives a large throughput regression because Marlin is a weight-only (W4A16) kernel and never reaches the FP4 tensor cores.

Do not pass --language-model-only — that flag belongs to the upstream text-only release and would disable the vision tower this build contains.

Transformers

Loading works (the quantization format is compressed-tensors), but transformers has no fast FP4 execution path — it dequantizes. Use vLLM or SGLang for anything performance-sensitive.

Usage

OpenAI-compatible API. This is a thinking model: it emits a reasoning block ending in </think>, so pass a generous max_tokens (≥256) and strip everything up to </think>, or disable thinking via chat_template_kwargs: {"enable_thinking": false}.

python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

r = client.chat.completions.create(
    model="kat-vl-flash",
    messages=[{"role": "user", "content": "Write a Python function to reverse a linked list."}],
    max_tokens=512, temperature=0.0,
)
print(r.choices[0].message.content.split("</think>")[-1].strip())

Sampling defaults follow upstream: temperature=1.0, top_p=0.95, top_k=20 for open-ended work, temperature=0.0 for deterministic coding.

Vision capability — READ THIS

The vision tower was never co-trained with this checkpoint's language model, which was heavily RL-tuned for agentic coding. Measured on the bf16 parent (this build inherits the behaviour, and 4-bit experts can only make it worse):

Table with columns: probe, bf16 parent
probe	bf16 parent
Dominant colour (4 solid-colour images)	4 / 4
Shape (circle / square / triangle)	3 / 3
Text/number reading (OCR)	0 / 4 — `CAT`→`CCT`, `42`→`48`, `HELLO`→`Hiro`

✅ Coarse visual questions: colour, shape, layout, rough scene gist.
❌ Not usable for: reading code from screenshots, OCR, UI labels, document or chart understanding — anything where one wrong character changes the answer.

If your workload is text-only, the vision tower costs you 0.9 GB and nothing else; the language model is what this checkpoint is for.

Acknowledgements

Kwaipilot/KAT-Coder-V2.5-Dev — base model and all language-model weights (Apache-2.0).
Qwen/Qwen3.6-35B-A3B — the architecture this model derives from (Apache-2.0).
The NVFP4 mixed-precision layout follows the approach popularised by Unsloth's Dynamic NVFP4 quants: keep the accuracy-critical layers at FP8/BF16, push the rest to W4A4 so Blackwell's FP4 tensor cores are actually used.

Released under Apache-2.0, matching both upstream models.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

beyoru

Model Tree

Base

beyoru/KAT-Coder-V2.5-Dev-VL

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality