JonaPoka

Qwen3.5-4B-Finnish

README

License: apache-2.0

What it is good at

Answering in Finnish without dropping back to English
Short EN→FI translation
Everyday drafts (cancellation email, messages) in a Finnish register
Wikipedia-style explanations after the native-text pass
Leading paraphrases for ice density and YK 1955 (after Stage 4)

What it is not

A 4B experimental fine-tune is not a native grammarian, not a historian, and not a Poro replacement. It still mixes Finnish cases on rules, and it still invents extra 1917 history. Prefer a 9B run on a 48 GB GPU if you need a serious Finnish assistant. Do not use this for legal, medical, or official advice.

How to run

Needs transformers>=4.57. This repo is the merged 4B checkpoint (~8.5 GB).

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "JonaPoka/Qwen3.5-4B-Finnish"

tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {
        "role": "system",
        "content": (
            "Olet avulias tekoälyavustaja. Vastaa suomeksi, ellei käyttäjä "
            "nimenomaan pyydä toista kieltä. Käytä luontevaa, kieliopillisesti oikeaa suomea."
        ),
    },
    {
        "role": "user",
        "content": "Selitä lyhyesti, miten suomen kielen partitiivi toimii. Anna kolme esimerkkiä.",
    },
]
enc = tok.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
    return_dict=True,
)
out = model.generate(
    **{k: v.to(model.device) for k, v in enc.items()},
    max_new_tokens=192,
    do_sample=False,
    repetition_penalty=1.08,
)
print(tok.decode(out[0][enc["input_ids"].shape[-1] :], skip_special_tokens=True))

If AutoModelForCausalLM fails to load the hybrid Qwen3.5 checkpoint, use AutoModelForImageTextToText with the same arguments. Stop on <|im_end|>. A light repetition penalty (≈1.08) reduces the ice-density loops we saw in greedy decode.

Training used one LoRA (rank 64) continued across four SFT stages, then merged into the base. You do not need peft at inference.

Training process

One LoRA, continued — not four separate models. Rank 64, alpha 128, dropout 0.05. Targets full-attention (q/k/v/o_proj), Gated DeltaNet (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj), and MLP (gate/up/down_proj). Sequence 1024, QLoRA 4-bit NF4, assistant-only loss, batch 4 × accum 4 (effective 16). Hardware: 1× RTX 3090 / 3090 Ti, 24 GB.

Stage 1 — Finnish lock (15k)

Table

Data	15,000-row quality slice of a 68k filtered mix: scored Poro-2 Finnish HQ, Turku paraphrase + FLORES, capped AutoIF-FI, Parsebank WebQA, tiny human chat
Recipe	`lr=1e-4`, 1 epoch, 938 steps in 4h 55m
Train loss	0.935 · token acc ~0.77
Why	Qwen3.5 has not been SFT’d on Finnish instruction data. This pass teaches “answer in Finnish.”

We did not dump the unfiltered 1.4M Poro-2 collection. Judge labels (HYÖDYLLINEN) are not native Finnish.

Stage 2 — GIGO filter, unused rows (13.7k)

Table

Data	13,672 rows the 15k run never saw, after dropping already-trained ids, `n_chars>2400`, markdown essays, English leftovers, calques, EN-target FLORES, non-Parsebank WebQA. Plus 10 handwritten gold dialogues ×12.
Recipe	continue the Stage 1 adapter, `lr=4e-5`, 1 epoch, 855 steps in 4h 39m
Train loss	0.958 · token acc ~0.76

Gold was ~1% of this mix, so morphology barely moved. Register and stopping did.

Stage 3 — new native text (11.2k), not a replay

Replaying the 68k would mostly memorize Llama-3.3 Finnish essays. Stage 3 continues the Stage 2 adapter on new licensed text converted to user→assistant turns (the gold is already in the source: Wikipedia lead, or the actual next paragraph).

Table with columns: Source, Rows, License, Task
Source	Rows	License	Task
Finnish Wikipedia (`Finnish-NLP/wikipedia_20231101_fi_cleaned`)	4,500 explain + 1,500 continue	CC BY-SA	“Selitä lyhyesti {title}” → lead; next-paragraph continuation
Project Gutenberg Finnish (`Varho/gutenberg-fi`)	4,000 continue + 800 extractive leads	Public domain (USA)	Continue the next paragraph
Finnish Wikinews (`Fumika/Wikinews-multilingual`, `lang=fi`)	110

Not included (would block a public Hub release): Yle and STT full-text archives (Kielipankki academic / no redistribution), Helsingin Sanomat, scraped commercial newsletters. Not used: raw continued pretraining on whole books (that strips instruction following).

Stage 4 — gold polish SFT (this checkpoint)

Continue the Stage 3 native LoRA. Config: configs/sft_qlora_4b_polish.yaml. No DPO after this.

Table

Data	75 handwritten gold dialogues ×48 (3,600) + 700 Turku paraphrase + 300 Finnish FLORES = 4,600 rows
Recipe	`lr=2e-5`, 1 epoch, 288 / 288 steps in ~55 min
Train loss	0.262 · token acc 0.956

This merged checkpoint contains Stages 1–4. The LoRA was trained, then baked into Qwen/Qwen3.5-4B.

Performance (smoke)

There is no public Finnish instruction leaderboard we trust for a 4B adapter. Evaluation is a fixed smoke set: the same prompts after every run, greedy decode, thinking off, max_new_tokens=192, repetition_penalty=1.08.

Table with columns: Prompt, Stage 1 (15k), Stage 2 (clean 13.7k), Stage 3 (native 11.2k), Stage 4 (gold polish)
Prompt	Stage 1 (15k)	Stage 2 (clean 13.7k)	Stage 3 (native 11.2k)	Stage 4 (gold polish)
Partitive	Allative `-lle` / genitive `-n` as “partitive”	Wrong rule (`-sta`+`-n`); kahvia / leipää	Still `-n`; kahvia / leipää kept	Examples kahvia / autoa / kirjaa are right; the stated rule still mentions `-n`
Dentist email

Stage 3 paraphrases already had ice-density and YK 1955 right; Stage 4 moved the open ice prompt off the invert, cleaned the dentist email, and made partitive examples usable. The partitive rule and the open 1917 prompt are still unreliable.

How to read this. Stage 1 solved language lock. Stage 2 improved register and stopping. Stage 3 added native-text style; leading factual prompts can work while open versions still fail. Stage 4 is a gold-heavy polish of that same LoRA, not a new model and not DPO.

Limitations

Case. Partitive examples improved at Stage 4; the verbal rule still confuses partitive with -n. Do not trust morphology explanations.
History. Independence Day 6 Dec 1917 is usually right; extra invented dates (1919, YK-as-independence) still appear on the open 1917 prompt.
Science. The leading ice-density paraphrase is correct; the open lake-ice explanation is clumsy and was inverted in earlier stages.
Capacity. Qwen’s own multilingual numbers are higher for 9B than 4B on every published set. Finnish morphology needs that headroom.
No DPO. Preference tuning was skipped because of a TRL / torch / transformers mismatch. The polish SFT already moved the demo prompts a lot.

Data licences

Base weights: Apache-2.0 (Qwen).
Stage 1–2 instruction rows keep their source licences. The Poro-2 HQ slice includes Llama-3.3-generated Finnish; Llama terms still apply to that slice if you redistribute it.
Stage 3: Wikipedia CC BY-SA, Gutenberg public domain (USA), Wikinews CC BY-2.5. Handwritten gold is provided for this fine-tune.
Stage 4: upsampled handwritten gold + Turku paraphrase + FLORES (same licences as the source rows).

Citation

markdown
@misc{qwen35-4b-finnish,
  title        = {Qwen3.5-4B Finnish (experimental)},
  author       = {JonaPoka},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/JonaPoka/Qwen3.5-4B-Finnish},
  note         = {Merged LoRA SFT on Qwen/Qwen3.5-4B, thinking disabled, no DPO}
}

Base model: Qwen Team, Qwen3.5.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

JonaPoka

Model Tree

Base

Qwen/Qwen3.5-4B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality