GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
KartiOS
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
KartiOS
Model Tree
KartiOS/Karti-Small-VL-4B
Input Modalities
Output Modalities
Supported Functionality
vLLM 0.27.1, 32k context, FP8 KV cache, temperature 0, single stream, best of 3.
| tokens | BF16 | NVFP4 |
|---|---|---|
| 128 | 21.0 tok/s | 51.7 tok/s |
| 512 | 21.1 tok/s | 51.8 tok/s |
| on disk | 9.3 GB | 3.7 GB |
| serving footprint | ~20 GB | ~8 GB |
It also degrades far less under contention than BF16, because it moves roughly a third of the bytes per token — on a box already serving other models, that matters more than the single-stream figure.
A community NVFP4 of this same base returned !!!!!!!! for every image —
including a solid red square — while passing arithmetic, tool calls and 43 tok/s,
at 668 downloads. It had quantized the vision tower: 25 ignored layers, none of
them vision.
This build holds out 98 vision modules; all 297 vision tensors stay unquantized, and every release is gated on real images, not text canaries.
markdown
PASS VISION solid red 'red' PASS arithmetic 19x23 437PASS VISION solid green 'green' PASS tool call parses ✓
Quantization costs accuracy. Same 603 rows, same endpoint, same decode path.
| base | NVFP4 | BF16 | |
|---|---|---|---|
| Invented-identifier rate | 0.378 | 0.048 | 0.023 |
| Panel reading | 0.595 | 0.924 | 0.967 |
~8× fewer invented identifiers than the untrained base; about half the BF16 margin given back.
Use NVFP4 when throughput or footprint is the constraint. Use BF16 when you need the best accuracy on reading identifiers exactly.
bash
vllm serve KartiOS/Karti-Small-VL-4B-NVFP4 \--max-model-len 32768 --kv-cache-dtype fp8 \--enable-auto-tool-choice --tool-call-parser qwen3_xml \--default-chat-template-kwargs '{"enable_thinking": false}'
Held out: lm_head · re:.*\.visual\..* (297 tensors) · re:^mtp\..* ·
re:.*linear_attn\.conv1d\.* — 147 entries, 248 modules quantized.
Calibration is frozen so version-over-version comparisons measure the
training, not the quantizer: lmms-lab/flickr30k test, 512 samples, seq 2048,
seed 115, multimodal — images through the model's own processor, never text-only.
It overlaps none of this program's evaluation sets.
No MTP head in this build, so no built-in speculative decoding; the BF16 has it. Serve the weights directly, never as a LoRA adapter over the base.
From Lumbridge.