What it is good at
- Bigger-model quality at 4B. Averaged over 238 languages, 47% of its questions pass every check: as many as
Qwen3.5-27B, and more than Llama 3.3 70B (43%), gpt-oss-20b (39%) and Qwen3.5-9B (42%). Only Gemma 4 31B scores
higher (51%).
- Strongest where other models are weakest. In languages with little training data it leads every open model
of its size and most larger ones: 31% usable in the rarest languages, against 15–25% for Qwen, gpt-oss and
Llama models up to 70B.
- Best at cross-lingual questions. Ask for English or French questions on a passage in another language, and
82% of them are usable, the highest of any model tested (Qwen3.5-27B 71%, Llama 3.3 70B 70%, Gemma 4 31B 68%).
- Questions that find their passage. Its questions retrieve their own passage more often than any other
model's (round-trip R@1 0.70), so they make strong query–passage training pairs.
- Small and fast. At 4B it runs comfortably on a single GPU with vLLM, so generating questions for millions of
passages is practical.
How to use it
The prompt is plain text, with no chat template. Write the language names out in full, in English, such as
"Spanish", "Japanese", "Yoruba" or "English".
Passage (<passage language>):<passage, up to ~2,000 characters> Questions in <question language> that this passage answers, in your own words:
The model answers with one question per line, each starting with - , and then stops.
vLLM (recommended)
from vllm import LLM, SamplingParams PROMPT = ("Passage ({lang}):\n{passage}\n\n" "Questions in {qlang} that this passage answers, in your own words:\n") llm = LLM(model="olaverse/MIST-QG-1.0", max_model_len=4096)params = SamplingParams(max_tokens=256, temperature=0.3, top_p=0.9, stop=["\n\n"]) passage = "Ọjà Ọ̀jọ̀ jẹ́ ọjà ìbílẹ̀ tó gbajúmọ̀ ní ìlú Ìbàdàn..."prompts = [PROMPT.format(lang="Yoruba", qlang="Yoruba", passage=passage), # same language PROMPT.format(lang="Yoruba", qlang="English", passage=passage)] # cross-lingual for out in llm.generate(prompts, params): questions = [l[2:].strip() for l in out.outputs[0].text.split("\n") if l.startswith("- ")] print(questions)
import torchfrom transformers import AutoModelForImageTextToText, AutoTokenizer tok = AutoTokenizer.from_pretrained("olaverse/MIST-QG-1.0")model = AutoModelForImageTextToText.from_pretrained("olaverse/MIST-QG-1.0", dtype=torch.bfloat16).cuda() prompt = ("Passage (Hausa):\n" + passage + "\n\n" "Questions in English that this passage answers, in your own words:\n")ids = tok(prompt, return_tensors="pt").input_ids.cuda()out = model.generate(ids, max_new_tokens=256, do_sample=True, temperature=0.3, top_p=0.9)print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
Qwen3.5 is a vision-language architecture, so the model loads with AutoModelForImageTextToText. MIST-QG uses
only its text side.
Tips
- Question languages for cross-lingual use: English or French, on a passage in any of the 324 languages.
- More questions per passage: same-language prompts usually give two to three questions. Cross-lingual prompts
give one, so sample several times with temperature 0.3 if you want more.
- Passage length: up to about 2,000 characters per prompt. Split longer documents into passages first.
Evaluation
All systems saw the same held-out passages, which were never used in training:
- Same language: 2,249 test passages in 238 languages, capped at 12 passages per language. Every African-language
test passage was kept.
- Cross-lingual: 340 passages in 153 languages, with questions requested in English (89%) or French (11%).
Each question is checked four ways, and it counts as usable only if it passes all four:
- Answerable. Qwen3.6-35B-A3B grades from 0 to 3 whether the passage answers the question; it must score at
least 2.
- Finds its passage. Qwen3-Embedding-8B, given only the question, must rank the question's own passage first
among all test passages in that language. For the cross-lingual test, the pool is all 340 cross-lingual passages.
- Not copied. Fewer than 60% of the question's word 3-grams may also appear in the passage.
- Right language. GlotLID must identify the language that was asked for.
The other models were given a plain instruction asking for up to six questions in the target language, written in
their own words. The "reference questions" are the dataset's own questions for the same passages. Every score is
averaged per language, so each language counts once, as in MTEB. Language groups follow how much MIST-QG
training data a language had: major (5,000+ passages, 90 test languages), mid-resource (1,000–5,000, 33),
low-resource (200–1,000, 32) and rare (under 200, 83).
Table with columns: Usable share, All languages, Major, Mid-resource, Low-resource, Rare, Cross-lingual| Usable share | All languages | Major | Mid-resource | Low-resource | Rare | Cross-lingual |
|---|
| MIST-QG-1.0 (4B) | 0.471 | 0.697 | 0.365 | 0.376 | 0.305 | 0.815 |
| Qwen3.5-4B | 0.385 | 0.671 | 0.296 | 0.243 | 0.166 | 0.595 |
| Qwen3-8B | 0.351 | 0.635 | 0.194 | 0.244 | 0.147 | 0.589 |
| Qwen3.5-9B | 0.420 | 0.686 | 0.336 | 0.298 | 0.210 | 0.653 |
| gpt-oss-20b | 0.393 | 0.654 | 0.322 | 0.318 | 0.162 | 0.638 |
| Qwen3.5-27B | 0.467 | 0.741 | 0.389 | 0.364 | 0.242 | 0.714 |
| Gemma 4 31B | 0.510 | 0.729 | 0.372 | 0.462 | 0.328 | 0.676 |
| Llama 3.3 70B | 0.429 | 0.668 | 0.318 | 0.345 | 0.247 | 0.705 |
| Reference questions | 0.525 | 0.710 | 0.405 | 0.409 | 0.417 | 0.904 |
Table with columns: Same language, all languages, Answerable, Finds its passage, Copied, Right language, Questions per passage| Same language, all languages | Answerable | Finds its passage | Copied | Right language | Questions per passage |
|---|
| MIST-QG-1.0 (4B) | 0.858 | 0.696 | 8.5% | 85.6% | 1.8 |
| Qwen3.5-4B | 0.788 | 0.682 | 16.7% | 79.5% | 5.9 |
| Qwen3-8B | 0.766 | 0.674 | 19.3% | 76.0% | 5.7 |
| Qwen3.5-9B | 0.799 | 0.677 | 9.3% | 80.0% | 6.0 |
| gpt-oss-20b | 0.741 | 0.678 | 5.7% | 75.1% | 5.9 |
| Qwen3.5-27B | 0.863 | 0.680 | 11.0% | 86.9% | 6.0 |
| Gemma 4 31B | 0.850 | 0.664 | 1.8% | 87.8% | 5.8 |
| Llama 3.3 70B | 0.803 | 0.684 | 12.0% | 86.1% | 6.0 |
On the cross-lingual test, MIST-QG-1.0's questions are 99.8% in the requested language, 0% copied, answerable
0.835 and find their passage 0.937.
Every figure is a share of questions, so it measures quality, not quantity: the other models write about six
questions per passage, MIST-QG about two.
Training
Stage 1, same language.
- Data: 991k passages in 324 languages, each paired with up to six questions in its own language. Sources:
WebFAQ, MQA, MIRACL, TyDi QA, parts of the KaLM fine-tuning collection, and LLM-written questions on FineWeb-2,
WURA and AfriCLIRMatrix passages.
- Filtering: only real questions were kept. Questions that lift a sentence from the passage were removed (tone
marks are ignored when matching), and duplicates were dropped.
- Run: one epoch of 7,746 steps at 128 passages per step. Learning rate 2e-5, cosine decay to 10%, 200 warm-up
steps.
Stage 2, cross-lingual.
- Data: 36.5k passages in about 300 languages, each with a question in English or French, mixed with 60k
same-language passages from stage 1 so the model keeps its same-language skill.
- Run: 755 steps, learning rate 1e-5, 30 warm-up steps.
Both stages:
- Loss: computed on the question tokens only.
- Precision and optimiser: bf16, gradient checkpointing, 8-bit AdamW (β = 0.9, 0.95).
- Sequence length: up to 1,024 tokens.
- Hardware: one A100 80 GB.
The test passages of both evaluations were held out from all training.
License
The model is released under Apache-2.0, the licence of its base model, Qwen3.5-4B-Base. The training data combines
sources under their own licences: Apache-2.0, CC-BY-4.0, CC-BY-SA (Wikipedia text) and ODC-BY (FineWeb-2).
Citation
@misc{mist-qg-1.0, title = {MIST-QG-1.0: multilingual question generation for retrieval training data}, author = {Olaverse}, year = {2026}, url = {https://huggingface.co/olaverse/MIST-QG-1.0}}