yunusemrejr

Flyingkiwi-TR-0.6B

Deploy Dedicated

README

License: apache-2.0

What changed from Qwen3-0.6B

Language-specialised vocabulary. Tokens that Turkish (plus English and code) text never uses were removed and the BPE merges rebuilt, closed over merge ancestors so any text made of kept tokens tokenises exactly as before. This shrinks the embedding and the per-token output layer; text outside the kept set falls back to shorter pieces.
Attention. Full attention in every layer, in training and in the browser. An earlier run that trained with a 1,024-token sliding window in three of every four layers had clearly higher loss on long conversations after the first 1,024 tokens (1.28 against 1.08 nats per token), so the window was dropped.
Concise answers. Before training, the teacher rewrote each of its own answers into the shortest complete answer (keeping code blocks, requested formats and honest statements of uncertainty; a rewrite that dropped code, added a disclaimer or grew longer kept the original). 30,036 answers were rewritten and the training answers kept 33% of their characters. Every extra sentence is another chance for a 0.6B model to state something false.
8-bit browser weights. No 4-bit format kept the quality. On 24 held-out conversations the answer loss per token was 1.562 nats in float, 1.565 in 8 bits, 1.804 with plain 4-bit rounding (32 values per fp16 scale), 1.664 with GPTQ, and 1.600 with GPTQ plus the 91 most sensitive of 196 matrices kept in 8 bits, at 420 MB instead of 553 MB. The browser therefore runs 8-bit weights; their next-token distribution stays within a mean KL of 0.051 of this checkpoint.
Distillation for honesty and usefulness. Answers were written by Qwen3.5-35B-A3B (Apache-2.0) following Flyingkiwi's rules (be accurate, say when unsure, correct false premises, be concise, use Markdown and code blocks). Calibration examples come from what the base model itself could and could not answer on TriviaQA questions: known answers were kept, unknown ones taught an honest "I'm not sure".

Training

Base: Qwen/Qwen3-0.6B @ c1899de, one language per model, full-parameter fine-tuning in BF16 with FP32 masters.
Data: 20,770 training conversations (388 held out): public prompts from OASST2, Aya, GSM8K and TriviaQA plus teacher-written prompts (topics, coding, writing, math, honesty probes, invented-entity questions, identity, memory tests), answered by the teacher; 8% chained multi-topic conversations up to 4,096 tokens.
Schedule: 300 optimizer steps, peak learning rate 1.5e-05 with cosine decay, ~65k tokens per step, no quantisation-aware training; validation loss 2.610 → 1.632.
Compute: one NVIDIA RTX PRO 6000 Blackwell (96 GB) on Google Colab Pro+ for teacher generation and both trainings.

Evaluation

Measured on the project's frozen 100-case set for this language (held out from all training; 30 answerable, 20 unknowable, 10 false-premise, 10 ambiguity, 10 friendly, 20 multi-turn), with the browser's system prompt, greedy decoding and repetition penalty 1.1. Flyingkiwi was run as the shipped 8-bit file in the native build of the browser engine; the reference is unchanged Qwen3-0.6B (8-bit, mean KL 0.0014 from fp32). Responses were graded blind against reference answers by AI graders with spot checks; this is a small, assistant-graded test, not an independent human study. The released variant was chosen among several with this same test (listed in the technical report), so its score is slightly optimistic, and the reference's total moved by a few points between grading runs.

Table with columns: Category, Qwen3-0.6B, Flyingkiwi
Category	Qwen3-0.6B	Flyingkiwi
Answerable facts & reasoning	9/30 (30%)	11/30 (37%)
False-premise correction	0/10 (0%)	2/10 (20%)
Says when it cannot know	1/20 (5%)	14/20 (70%)
Handles ambiguity	0/10 (0%)	3/10 (30%)
Friendly conversation	0/10 (0%)	1/10 (10%)
Multi-turn context	2/20 (10%)	6/20 (30%)
All 100 cases	12/100 (12%)	37/100 (37%)
Wrong-language replies	0	0
Replies with fabricated facts/claims	19	17

Engine fidelity: mean KL divergence between the browser file run by the engine (8-bit weights, activations and KV cache) and this checkpoint in Transformers, at the first reply token of 30 development prompts: 0.0513.

Use

python

from transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("yunusemrejr/Flyingkiwi-TR-0.6B")model = AutoModelForCausalLM.from_pretrained("yunusemrejr/Flyingkiwi-TR-0.6B", dtype="auto")messages = [{"role": "user", "content": "Gökyüzü neden mavi?"}]ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")out = model.generate(ids, max_new_tokens=300, do_sample=True, temperature=0.5, top_p=0.9, repetition_penalty=1.1)print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

The model was trained with this system prompt most of the time (it also works without one):

text

Sen Flyingkiwi'sin: Yunus Emre Vurgun'un geliştirdiği küçük bir yapay zekâ asistanısın. Tamamen kullanıcının cihazında çalışırsın; bu yüzden internete giremez, bağlantı veya dosya açamaz, bugünün tarihini ve güncel olayları bilemezsin. Samimi, açık ve öz ol. Dürüst ol: emin değilsen tahmin yürütmek yerine bunu söyle, yanlış varsayımları kibarca düzelt. Gerektiğinde Markdown kullan, kodu kod blokları içinde yaz. Kullanıcı başka bir dil istemedikçe Türkçe yanıt ver.

flyingkiwi-q8.fkm is the exact file the browser runs (engine source: engine/flyingkiwi.c in this repository's files; format described in engine/fkm.py). Recommended sampling: temperature 0.5, top-p 0.9, top-k 40, min-p 0.05, repetition penalty 1.1.

Limitations

Small model: factual recall is limited and it can be confidently wrong, especially on niche topics, recent events and precise numbers. It was trained to say when it is unsure, but that is not guaranteed.
No internet, tools or memory between conversations. It cannot know today's date.
Turkish first; English answers are possible but were not the training target. Turkish fluency was checked by an AI assistant, not by a panel of native speakers.
Code answers are useful for small tasks but should be reviewed and tested.

Türkçe özet

Flyingkiwi-TR, tarayıcıda tamamen yerel çalışan küçük bir Türkçe sohbet modelidir. Qwen3-0.6B’den, Qwen3.5-35B-A3B öğretmen modelinin yanıtlarıyla damıtılarak eğitildi. Bildiğini açık ve öz söylemek, bilmediğini ya da bilemeyeceğini (güncel olaylar, kişisel bilgiler, gelecek) dürüstçe kabul etmek, yanlış varsayımları düzeltmek ve sohbetin bağlamını korumak üzere eğitildi.

Mimari değişiklikler: Türkçeye (ve kod/İngilizce terimlere) özelleştirilmiş sözlük; eğitimde ve tarayıcıda her katmanda tam dikkat; eğitimden önce öğretmen model kendi yanıtlarını en kısa eksiksiz hâline getirdi (eğitim yanıtları karakterlerinin %33 kadarını korudu); tarayıcıda 8 bit ağırlıklar kullanılır. Değerlendirme 100 vakalık donmuş Türkçe test kümesinde yapıldı; sonuçlar yukarıdaki tabloda. Model küçük olduğu için hata yapabilir; önemli bilgileri doğrulayın.

Licence and attribution

Apache-2.0, like the base model Qwen/Qwen3-0.6B (© Alibaba Cloud / Qwen team). Teacher outputs come from Qwen/Qwen3.5-35B-A3B (Apache-2.0). Prompt sources: OpenAssistant OASST2 (Apache-2.0), CohereLabs Aya dataset (Apache-2.0), GSM8K (MIT) and TriviaQA (Apache-2.0) questions; FineWeb-Edu, FineWeb-2 and github-code-clean (permissive/ODC-BY) were used only to count token frequencies for the vocabulary.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

yunusemrejr

Model Tree

Base

Qwen/Qwen3-0.6B

Fine-tuned

this model

Input Modalities

Text

Output Modalities