issai

Qwen3.5-9B-Kazakh

README

License: apache-2.0

Methodology

This model is produced in two stages on top of our Kazakh continued-pretraining model issai/Qwen3.5-9B-Base-Kazakh, the official Qwen/Qwen3.5-9B-Base continued-pretrained on Kazakh text with a +16K Kazakh-token vocabulary extension. No additional post-training (SFT, RLHF, or RLVR) is used.

1. Chat-vector merge, transferring instruct alignment

We transfer the instruction-following / chat alignment of the official post-trained model using the chat vector method (Huang et al., 2024). Let

θ_inst: weights of the official post-trained Qwen/Qwen3.5-9B
θ_base: weights of the official base Qwen/Qwen3.5-9B-Base
θ_cpt: weights of our Kazakh base model issai/Qwen3.5-9B-Base-Kazakh

The chat vector is the task vector of post-training,

$τ chat = θ inst - θ base ,$

and it is added to our Kazakh continued-pretraining model:

$θ merged = θ cpt + τ chat = θ cpt $

applied per parameter tensor, by name. Because our tokenizer is vocabulary-extended, the input-embedding and output (lm_head) matrices are merged row-wise: the shared original-vocabulary rows receive the chat vector, while the 16K new Kazakh-token rows are kept from θ_cpt (the official models have no counterpart for them). embed_tokens and lm_head are untied and are merged independently.

Special-token pinning. The rows of the format-control special tokens <think>, </think>, <|im_start|>, <|im_end|> and <|endoftext|> (in both the embedding and lm_head) are set to the official post-trained model's exact values instead of the merged values. These few rows drive the chat template's control flow (emitting </think> to close the reasoning block and <|im_end|> to end the turn). The raw chat vector otherwise detunes them and breaks this template logic, causing the model to loop and never terminate. Pinning them restores reliable termination while preserving the Kazakh knowledge acquired during continued pretraining.

2. Vision re-integration, restoring multimodality

Because our continued pretraining was performed on text only, with the vision branch removed beforehand, the chat-vector merge operates on the language model and produces a text-only model. To restore image and video understanding, the vision encoder and multimodal projector from the official Qwen/Qwen3.5-9B are re-attached to this Kazakh-adapted, instruct-aligned language backbone, and the architecture is set back to Qwen3_5ForConditionalGeneration. The result is a multimodal model that largely preserves the official model's vision capabilities while its language model handles Kazakh (and English) with instruct-style alignment and correct <think>...</think> reasoning.

Evaluation

We evaluate Qwen3.5-9B-Kazakh against its base model Qwen3.5-9B on Kazakh and English benchmarks under identical settings (thinking mode enabled):

temperature = 1.0
top_p = 0.95
top_k = 20
presence_penalty = 1.5
max_tokens = 81920

Scores are accuracy (%).

Kazakh

The Kazakh benchmarks (most of which are available in the ISSAI Qolda suite: language, vision) use Kazakh questions and answer choices. KazMMLU and KazCulture are native Kazakh benchmarks; the rest are Kazakh adaptations of the standard benchmarks.

English

Model usage

The following package versions are used:

transformers>=5.12.1
a recent torch build (matching your CUDA)
vllm>=0.24.0 (for serving)
optional: flash-linear-attention and causal-conv1d for faster linear-attention inference

The model reasons by default, emitting a chain-of-thought inside <think> ... </think>.

Transformers

python
from transformers import AutoProcessor, AutoModelForMultimodalLM

model_id = "issai/Qwen3.5-9B-Kazakh"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

messages = [
    {"role": "user", "content": [{"type": "text", "text": "Балқаш көлінің қандай ерекшелігі бар?"}]}
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs, max_new_tokens=32768,
    do_sample=True, temperature=1.0, top_p=0.95, top_k=20,
)

print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Image input:

python
messages = [
    {"role": "user", "content": [
        {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
        {"type": "text", "text": "Суретте не бейнеленген?"},
    ]}
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs, max_new_tokens=32768,
    do_sample=True, temperature=1.0, top_p=0.95, top_k=20,
)

print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

To disable thinking, pass enable_thinking=False to apply_chat_template.

vLLM

Serve an OpenAI-compatible endpoint:

bash
vllm serve issai/Qwen3.5-9B-Kazakh \
    --served-model-name qwen \
    --reasoning-parser qwen3 \
    --max-model-len 90000

python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": """Төмендегі экзопланеталардың қайсысында тығыздығы ең жоғары?
a) Жермен бірдей құрамы бар, бірақ Жерден 5 есе ауыр планета.
b) Жермен бірдей құрамды, бірақ Жерден екі есе жеңіл планета.
c) Массасы Жерден 2 есе үлкен және тығыздығы шамамен 5.5 г/см^3 болатын планета.
d) Жердің массасы мен радиусымен тең планета."""}],
    max_tokens=81920, temperature=1.0, top_p=0.95, presence_penalty=1.5,
    extra_body={"top_k": 20},
)

print(resp.choices[0].message.reasoning)          # chain-of-thought
print(resp.choices[0].message.content)            # final answer

Image input:

python
import base64

image = base64.b64encode(open("image.png", "rb").read()).decode()

resp = client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}},
        {"type": "text", "text": "Суретте қандай фигуралар бар?"},
    ]}],
    max_tokens=81920, temperature=1.0, top_p=0.95, presence_penalty=1.5,
    extra_body={"top_k": 20},
)

print(resp.choices[0].message.content)

To disable thinking, add "chat_template_kwargs": {"enable_thinking": false} to extra_body.

License

This model is released under the Apache 2.0 license, inherited from the original Qwen3.5 model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

issai

Model Tree

Base

issai/Qwen3.5-9B-Base-Kazakh

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Methodology

1. Chat-vector merge, transferring instruct alignment

We transfer the instruction-following / chat alignment of the official post-trained model using the chat vector method (Huang et al., 2024). Let

θ_inst: weights of the official post-trained Qwen/Qwen3.5-9B
θ_base: weights of the official base Qwen/Qwen3.5-9B-Base
θ_cpt: weights of our Kazakh base model issai/Qwen3.5-9B-Base-Kazakh

The chat vector is the task vector of post-training,

$τ chat = θ inst - θ base ,$

and it is added to our Kazakh continued-pretraining model:

$θ merged = θ cpt + τ chat = θ cpt $

2. Vision re-integration, restoring multimodality

Evaluation

We evaluate Qwen3.5-9B-Kazakh against its base model Qwen3.5-9B on Kazakh and English benchmarks under identical settings (thinking mode enabled):

temperature = 1.0
top_p = 0.95
top_k = 20
presence_penalty = 1.5
max_tokens = 81920

Scores are accuracy (%).

Kazakh

English

Model usage

The following package versions are used:

transformers>=5.12.1
a recent torch build (matching your CUDA)
vllm>=0.24.0 (for serving)
optional: flash-linear-attention and causal-conv1d for faster linear-attention inference

The model reasons by default, emitting a chain-of-thought inside <think> ... </think>.

Transformers

python
from transformers import AutoProcessor, AutoModelForMultimodalLM

model_id = "issai/Qwen3.5-9B-Kazakh"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

messages = [
    {"role": "user", "content": [{"type": "text", "text": "Балқаш көлінің қандай ерекшелігі бар?"}]}
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs, max_new_tokens=32768,
    do_sample=True, temperature=1.0, top_p=0.95, top_k=20,
)

print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Image input:

python
messages = [
    {"role": "user", "content": [
        {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
        {"type": "text", "text": "Суретте не бейнеленген?"},
    ]}
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs, max_new_tokens=32768,
    do_sample=True, temperature=1.0, top_p=0.95, top_k=20,
)

print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

To disable thinking, pass enable_thinking=False to apply_chat_template.

vLLM

Serve an OpenAI-compatible endpoint:

bash
vllm serve issai/Qwen3.5-9B-Kazakh \
    --served-model-name qwen \
    --reasoning-parser qwen3 \
    --max-model-len 90000

python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": """Төмендегі экзопланеталардың қайсысында тығыздығы ең жоғары?
a) Жермен бірдей құрамы бар, бірақ Жерден 5 есе ауыр планета.
b) Жермен бірдей құрамды, бірақ Жерден екі есе жеңіл планета.
c) Массасы Жерден 2 есе үлкен және тығыздығы шамамен 5.5 г/см^3 болатын планета.
d) Жердің массасы мен радиусымен тең планета."""}],
    max_tokens=81920, temperature=1.0, top_p=0.95, presence_penalty=1.5,
    extra_body={"top_k": 20},
)

print(resp.choices[0].message.reasoning)          # chain-of-thought
print(resp.choices[0].message.content)            # final answer

Image input:

python
import base64

image = base64.b64encode(open("image.png", "rb").read()).decode()

resp = client.chat.completions.create(
    model="qwen",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}},
        {"type": "text", "text": "Суретте қандай фигуралар бар?"},
    ]}],
    max_tokens=81920, temperature=1.0, top_p=0.95, presence_penalty=1.5,
    extra_body={"top_k": 20},
)

print(resp.choices[0].message.content)

To disable thinking, add "chat_template_kwargs": {"enable_thinking": false} to extra_body.

License

This model is released under the Apache 2.0 license, inherited from the original Qwen3.5 model.