Model details
Table | |
|---|
| type | chat model with thinking (<think> … </think>), text and image input |
| architecture | Qwen3.5: 32-layer hybrid language model (24 Gated DeltaNet linear-attention + 8 full-attention layers) + Qwen3.5's vision encoder |
| parameters | ~4.6 B (language model 4.25 B) |
| vocabulary | 264,077 tokens: Qwen3.5's 248,077 + 16,000 new Kazakh/Russian tokens |
| precision | bfloat16 |
| languages | Kazakh, Russian, English |
| drop-in for | Qwen/Qwen3.5-4B: same architecture, chat template and usage |
Implementation
No training in this step: weights are only added, subtracted and copied.
Qwen3.5-4B-Base (B) ──┐Qwen3.5-4B (I) ──┼─► 1. chat vector: I + (C − B)our base model (C) ──┘ │ ▼ 2. pin 7 special-token rows to I ▼ 3. attach Qwen3.5-4B's vision encoder ▼ Qwen3.5-4B-Kazakh-OptimizedTokenizer
1. Chat vector merge
With B = Qwen3.5-4B-Base, I = Qwen3.5-4B
(Qwen's chat model) and C = our Kazakh base model:
C − B is what continued pretraining added: Kazakh and Russian knowledge and the new tokens. Adding it to Qwen's chat
model gives a chat model that knows Kazakh. This is applied to every weight of the language model.
The 16,000 new token rows exist only in C and are copied from C.
2. Special-token pinning
Continued pretraining on plain documents moves the embeddings of the chat-control tokens. These few rows decide when
the model closes its reasoning and ends its turn. Seven rows are therefore set back to Qwen3.5-4B's exact values:
Table with columns: tokens, why| tokens | why |
|---|
| `< | endoftext |
| `< | vision_start |
3. Vision encoder
Continued pretraining was text-only, so Qwen3.5-4B's vision encoder is attached back unchanged. The text part stays
bit-for-bit as merged.
Comparison
Three models: the original Qwen3.5-4B, ISSAI's earlier
Qwen3.5-4B-Kazakh, and this model.
At a glance
Table with columns: Qwen3.5-4B, issai/Qwen3.5-4B-Kazakh, this model | Qwen3.5-4B | issai/Qwen3.5-4B-Kazakh | this model |
|---|
| built on | — | issai/Qwen3.5-4B-Base-Kazakh | issai/Qwen3.5-4B-Base-Kazakh-OptimizedTokenizer |
| tokens per word, Kazakh / Russian / English | 3.48 / 1.89 / 1.25 | 1.83 / 2.23 / 1.25 | 1.76 / 1.72 / 1.25 |
| pinned special tokens | — | 5 | 7 |
| Kazakh text average (9 benchmarks) | 80.19 | 80.94 | 82.08 |
| English text average (6 benchmarks) | 92.38 | 91.47 | 91.72 |
Tokenizer efficiency
Tokens per word on the 488 Belebele (FLORES) passages of each language. Lower is better: faster answers, and more
text fits in the context.
Table with columns: language, Qwen3.5-4B, issai/Qwen3.5-4B-Kazakh, this model| language | Qwen3.5-4B | issai/Qwen3.5-4B-Kazakh | this model |
|---|
| Kazakh | 3.48 | 1.83 (1.90× fewer) | 1.76 (1.98× fewer) |
| Russian | 1.89 | 2.23 (1.18× more) | 1.72 (1.10× fewer) |
| English | 1.25 | 1.25 | 1.25 |
Example (Russian): Бескрайняя казахская степь тянется до самого горизонта, и в тихом ауле слышно, как поёт домбра.
is 31 tokens with Qwen3.5, 34 with the earlier ISSAI tokenizer, and 29 with this one.
Text benchmarks
Thinking on, full test sets, ISSAI's evaluation harness (vLLM), temperature 1.0, top-p 0.95, top-k 20, presence
penalty 1.5, up to 81,920 tokens. Kazakh benchmarks are from ISSAI's
Qolda language suite; KazMMLU is native Kazakh.
Accuracy in %.
Table with columns: benchmark, lang, n, Qwen3.5-4B, issai/Qwen3.5-4B-Kazakh, this model| benchmark | lang | n | Qwen3.5-4B | issai/Qwen3.5-4B-Kazakh | this model |
|---|
| KazMMLU | kk | 9,870 | 72.5 | 78.5 | 78.4 |
| MMLU | kk | 14,042 | 77.7 | 78.5 | 79.2 |
| MMLU-Pro | kk | 12,032 | 70.0 | 69.7 | 70.0 |
| MMLU-Redux | kk | 5,700 | 80.6 | 80.7 | 81.1 |
| GPQA | kk | 1,192 | 61.7 | 60.5 | 62.1 |
| GSM8K | kk | 1,319 | 88.0 | 89.0 | 90.3 |
| Belebele | kk | 900 | 84.6 | 85.6 | 87.2 |
| ARC-Challenge | kk | 1,172 | 91.3 | 91.8 | 93.9 |
| ARC-Easy | kk | 2,376 | 95.3 | 94.2 | 96.5 |
| Kazakh average | | | 80.19 | 80.94 | 82.08 |
| MMLU | en | 14,042 | 85.9 | 84.2 | 84.5 |
| MMLU-Pro | en | 12,032 | 79.4 | 77.4 | 77.6 |
| GSM8K | en | 1,319 | 96.2 | 95.8 | 96.1 |
| Belebele | en | 900 | 97.2 | 97.0 | 97.1 |
| ARC-Challenge | en | 1,172 | 96.9 | 95.8 | 96.3 |
| ARC-Easy | en | 2,376 | 98.7 | 98.6 | 98.7 |
| English average | | | 92.38 | 91.47 | 91.72 |
Qwen3.5-4B and this model: our runs. issai/Qwen3.5-4B-Kazakh: from its model card,
which used the same harness and settings. Its Qwen3.5-4B numbers match ours (GSM8K-kk: 87.9 there, 88.0 here).
Vision benchmarks
Rule-scored benchmarks, our runs of both models with the same harness and settings.
Table with columns: benchmark, lang, n, Qwen3.5-4B, this model| benchmark | lang | n | Qwen3.5-4B | this model |
|---|
| MMBench | kk | 4,329 | 87.3 | 88.5 |
| OCRBench | kk | 441 | 31.5 | 32.0 |
| AI2D | kk | 3,088 | 78.2 | 79.9 |
| MMStar | kk | 1,499 | 71.4 | 70.7 |
| MMStar | ru | 1,500 | 62.4 | 62.2 |
| MMBench | en | 4,377 | 89.7 | 89.6 |
| OCRBench | en | 1,000 | 82.3 | 82.0 |
| AI2D | en | 3,088 | 87.5 | 87.4 |
| MMStar | en | 1,500 | 77.1 | 75.2 |
Benchmarks scored by an LLM judge (RealWorldQA, MathVista, MathVision) are not included here.
What it shows
- Kazakh: +1.9 points over Qwen3.5-4B and +1.1 over the earlier ISSAI model on average; best or tied on 8 of 9
benchmarks (KazMMLU: 78.4 vs 78.5).
- English: within 0.7 points of Qwen3.5-4B on average, and slightly above the earlier ISSAI model.
- Vision: on par with Qwen3.5-4B: ahead on Kazakh MMBench, AI2D and OCRBench, within 2 points elsewhere.
How to use
Requires transformers 5.x (tested with 5.12.1). The model thinks by default (<think> … </think> before the
answer).
from transformers import AutoProcessor, AutoModelForImageTextToText name = "issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer"processor = AutoProcessor.from_pretrained(name)model = AutoModelForImageTextToText.from_pretrained(name, dtype="bfloat16", device_map="auto") messages = [{"role": "user", "content": [{"type": "text", "text": "Қазақ даласының ерекшелігі неде?"}]}]inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=8192, do_sample=True, temperature=0.6, top_p=0.95, top_k=20)print(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
- Image input: add
{"type": "image", "url": "<image url or path>"} before the text in content.
- Without thinking: pass
enable_thinking=False to apply_chat_template.
- vLLM:
vllm serve issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer --reasoning-parser qwen3
Limitations
- Not measured yet: Russian text benchmarks, KazCulture, PolyMath, English GPQA.
- No additional safety tuning beyond what the merge inherits from Qwen3.5-4B.
- The vision encoder was not trained on Kazakh data. Reading Kazakh text in images (OCRBench-kk ≈ 32%) is much weaker
than in English (≈ 82%).
Contributions
- Almubdi Mutaikhan: base model, chat-vector merge, special-token pinning, vision, evaluation, release.
- Batyr Arystanbekov: ISSAI's training pipeline and data used for the base model.
- Huseyin Atakan Varol: supervision.
License
Apache-2.0, the same as Qwen/Qwen3.5-4B.
Citation
@misc{mutaikhan2026qwen35kazakhopt, title = {Qwen3.5-4B-Kazakh-OptimizedTokenizer}, author = {Mutaikhan, Almubdi and Arystanbekov, Batyr and Varol, Huseyin Atakan}, year = {2026}, publisher = {Hugging Face}, howpublished = {\url{https://huggingface.co/issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer}}}