What the chat vector does, and what it does not
The merge is one operation and requires no instruction data:
Δ_chat = θ_instruct − θ_base # Qwen3-30B-A3B-Thinking-2507 − Qwen3-30B-A3B-Base
θ_merged = θ_cpt + Δ_chat # added to the continually pre-trained checkpoint
It installs instruction following and alignment for free. It also carries over
the donor's reasoning language, which continual pre-training does not
override. On a translated GSM8K probe, under a system prompt that explicitly asks
for Hong Kong Cantonese, this model opens its trace in Cantonese:
首先，用戶嘅問題係關於Janet嘅鴨蛋生意。我需要用香港廣東話回應，因為系統設定係CantoneseLLM。
(First, the user's question is about Janet's duck egg business. I need to respond in Hong Kong Cantonese, because the system is set to CantoneseLLM.)
…and then slides into Written Chinese later in the same span, ending up
discussing the output language rather than thinking in it:
…現在，用香港廣東話回應。…在回應中，要用口語廣東話。
(…Now, respond in Hong Kong Cantonese. … In the response, colloquial Cantonese should be used.)
現在 jin6 zoi6 is the Written Chinese form where Cantonese uses 而家 ji4 gaa1,
and 在⋯中 zoi6…zung1 is a Mandarin-shared construction. The final answer is
correct and reads as Cantonese; the reasoning that produced it does not.
Trace length over eight fixed probes (mean tokens between the reasoning tags):
Table with columns: Checkpoint, Mean CoT tokens, CoT-to-answer ratio| Checkpoint | Mean CoT tokens | CoT-to-answer ratio |
|---|
| Qwen3-30B-A3B-Thinking-2507 (donor) | 1,042 | 6.25 |
| This model | 1,442 | 5.85 |
| CantoneseLLM-v2.0-30B-A3B-Thinking (final) | 155 | 1.43 |
This model reasons at length — longer than its own donor. That is worth knowing:
the length was never the problem, the language was.
Benchmark results
HKCanto-Eval (Cheng et al., 2025),
reasoning mode on:
Table with columns: Model, MMLU, CantoMMLU, Cultural, Linguistic, Academic & Prof., Avg.| Model | MMLU | CantoMMLU | Cultural | Linguistic | Academic & Prof. | Avg. |
|---|
| Qwen3-30B-A3B-Thinking-2507 (donor) | 86.65 | 82.52 | 68.65 | 57.00 | 86.70 | 76.30 |
| This model | 80.71 | 80.26 | 70.24 | 55.00 | 85.59 |
The merge costs 1.94 points against the donor (−2.55%), and the regression is
concentrated in MMLU in English, which drops 5.94 points. Cultural is the one
category that improves — 68.65 → 70.24 — which is the continual pre-training
showing through, since that is the category most about Hong Kong.
This model scores 1.20 points above the final RLVR model. That is not a
reason to prefer it. The benchmark is multiple-choice and does not measure the
language of the chain-of-thought, which is the entire difference between the two
checkpoints. Selecting on this average gets you a model that thinks in Written
Chinese.
Artefacts released with this work
The CPT corpus itself is not released, but its two largest public constituents
are.
Table with columns: What it is | What it is |
|---|
| 🌐 Traditional-Chinese-Common-Crawl-by-year | Traditional Chinese extracted from all 111 Common Crawl snapshots, released per snapshot across thirteen years |
| 🇭🇰 Cantonese-Web-Data | The Cantonese subset of the above, filtered with CantoneseDetect and globally deduplicated to 477,298 unique documents |
| 🏋️ cantonese-nemo-gym-environments | The six NeMo-Gym environments and the multiplicative language reward. Used by the final models, not by this checkpoint; MIT licensed |
| 📊 |
Other models in this release
All four checkpoints are in the
CantoneseLLM v2.0 collection.
Table with columns: Model, What it is| Model | What it is |
|---|
| CantoneseLLM-v2.0-30B-A3B-Thinking-Chat-Vector-Merged | this model — CPT + chat-vector merge, before SFT, DPO and RLVR |
| CantoneseLLM-v2.0-30B-A3B-Thinking | The flagship, built from this checkpoint through SFT, DPO and two-stage RLVR. Reasons in Cantonese. Prefer it unless you specifically need this baseline |
| CantoneseLLM-v2.0-8B-Thinking-Chat-Vector-Merged | The same intermediate stage at 8B, where the inherited reasoning language is Simplified Chinese rather than Written Chinese |
| CantoneseLLM-v2.0-8B-Thinking | The 8B dense model through the full five stages |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "hon9kon9ize/CantoneseLLM-v2.0-30B-A3B-Thinking-Chat-Vector-Merged"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")
SYSTEM = "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"
messages = [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": "小明有 5 個蘋果，佢俾咗 2 個朋友，每人 1 個，跟住又買多 3 個。佢而家有幾多個蘋果？"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=2048, temperature=0.6, top_p=0.95)
print(tokenizer.decode(out[0][len(inputs.input_ids[0]):], skip_special_tokens=True))
vLLM
The stock command works — no special flags are required:
vllm serve hon9kon9ize/CantoneseLLM-v2.0-30B-A3B-Thinking-Chat-Vector-Merged
Add --tensor-parallel-size N to shard across N GPUs, and
--reasoning-parser qwen3 if you want the reasoning block returned separately
as reasoning_content rather than inline in content (parser names vary by
vLLM version).
OpenAI-compatible API
The served endpoint speaks the OpenAI protocol, so the official client works
unchanged:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
SYSTEM = "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"
resp = client.chat.completions.create(
model="hon9kon9ize/CantoneseLLM-v2.0-30B-A3B-Thinking-Chat-Vector-Merged",
messages=[
{"role": "system", "content": SYSTEM},
{"role": "user", "content": "小明有 5 個蘋果，佢俾咗 2 個朋友，每人 1 個，跟住又買多 3 個。佢而家有幾多個蘋果？"},
],
temperature=0.6,
top_p=0.95,
max_tokens=2048,
)
msg = resp.choices[0].message
print(getattr(msg, "reasoning_content", None) or "")
print(msg.content)
Or with curl:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "hon9kon9ize/CantoneseLLM-v2.0-30B-A3B-Thinking-Chat-Vector-Merged",
"messages": [
{"role": "system", "content": "你係CantoneseLLM，一個由Hon9Kon9ize開發嘅語言模型，請使用香港嘅廣東話回答用家問題"},
{"role": "user", "content": "點解香港嘅雨季集中喺五月到九月？"}
],
"temperature": 0.6, "top_p": 0.95, "max_tokens": 2048
}'
Sampling: temperature 0.6, top-p 0.95 — the settings used for every
evaluation reported above and in the paper. Avoid greedy decoding.
System prompt: the Cantonese system prompt above was used throughout
evaluation. Behaviour with other system prompts, or with none, is not
characterised.
Chat template and thinking mode are inherited from the donor. The chat vector
came from Qwen3-30B-A3B-Thinking-2507, a reasoning-only model, so this checkpoint
always produces a reasoning block. Do not expect /no_think to work as it does
in the stock Qwen3 hybrid models.
Training pipeline
Table with columns: Stage, What it installed, Compute| Stage | What it installed | Compute |
|---|
| Continual pre-training | Hong Kong knowledge and Cantonese lexis. 784M tokens over 568K rows, 530 steps (2.19B tokens seen, 2.79 passes), LR 1.5×10⁻⁵, on 64 TPU v6e chips (Google TRC) via MaxText | 199 TPU chip-hours |
| Chat-vector merge | Instruction following and alignment. Δ = Qwen3-30B-A3B-Thinking-2507 − Qwen3-30B-A3B-Base, added to the CPT checkpoint | 0 |
That is the whole pipeline for this checkpoint — no SFT, no DPO, no RLVR. The
corpus was 32.9% Common Crawl (5.4% of the total in Cantonese), 20.9% English
replay from Nemotron pre-training data, and 17.1% web fiction, which is the only
source where Cantonese is sustained over tens of thousands of tokens.
The CPT learning rate was selected across eight settings from 1.0×10⁻⁵ to
5.0×10⁻⁵ by benchmark average rather than by training loss — the paper reports
that loss anti-correlates with the quantity of interest in this regime.
Risks & Limitations
- It does not reason in Cantonese. This is the defining limitation and the
reason the rest of the pipeline exists. The trace opens in Cantonese and drifts
into Written Chinese. If you need Cantonese chain-of-thought, use
the final model.
- Instruction following is inherited, not trained. It arrives through weight
arithmetic from the donor and was never fine-tuned on Cantonese instruction
data. Behaviour outside the donor's own distribution is not characterised.
- MMLU regression. 5.94 points below the donor in English MMLU. The merge is
not free in capability terms even though it is free in compute terms.
- No translation, data-curation or judging ability. Those were installed at
SFT, which this checkpoint precedes.
- CPT was 784M tokens — small by continual-pre-training standards (Taiwan-LLM
used 35.1B, Swallow and SEA-LION 200B each). Hong Kong knowledge is real but
thin, and the corpus leans on web fiction for long-form Cantonese.
- Alignment is whatever the chat vector carried over. No additional safety
tuning was performed at any stage.
- Long context is inherited and untested — the CPT sequence length was 4,096
packed. Behaviour beyond that is whatever the Qwen3 base provides.
- Standard LLM caveats apply: it will hallucinate.
Citation
@misc{cantonesellm_v2,
title={CantoneseLLM v2: Reasoning in a Low-Resource Language},
author={Tsz Chung Cheng and Chung Shing Cheng and Chaak Ming Lau and Cheuk Hei Chong},
year={2026},
eprint={2609.06970},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2609.06970},
}
The chat vector method is due to Huang et al., Chat Vector: A Simple Approach
to Equip LLMs with Instruction Following and Model Alignment in New Languages,
ACL 2024, pp. 10943–10959.
Acknowledgements
Continual pre-training (CPT) was carried out on Cloud TPUs (Tensor Processing Units) from Google's TPU Research Cloud with MaxText. Post-training was carried out on computer resources offered under the category of General Projects by Research Institute for Information Technology, Kyushu University. Usage fee and cost of data-curation costs with proprietary APIs were covered by Votee AI
The RLVR stage builds on NVIDIA's NeMo-RL and NeMo-Gym, and on the Nemotron
post-training and RL datasets.
