What it does
Made-up results, real output (the demo in searchreader.py):
Table with columns: question, results given, answer| question | results given | answer |
|---|
| When does the Harbor Street library open on Saturday? | [1] Harbor Street Library - Hours and location: The Harbor Street Library is open Monday to Friday from 8 am to 8 pm. On Saturday the library opens at 10 am and closes at 4 pm. It is closed on Sunday. [2] Harbor Street Library events: Story time for children runs every Wednesday morning in the reading room. [3] City libraries | Parking: Free parking is available behind the Harbor Street Library for up to two hours. | 10 am |
| Who won the 2031 Harbor Street Chess Open? | [1] Harbor Street Chess Open 2025 results: Mara Lindqvist won the 2025 Harbor Street Chess Open with 7 points from 9 games. [2] Harbor Street Chess Open 2026: The 2026 Harbor Street Chess Open was won by Tomas Varga after a tie-break against Mara Lindqvist. [3] About the Harbor Street Chess Open: The open has been played every June at the Harbor Street Library since 2019. | not found |
Asked "Who won the 2026 Harbor Street Chess Open?" with the same results, it answers Tomas Varga. Given only results
[1] and [2], it also answers Tomas Varga to the 2031 question, so not found is not guaranteed for a future year when a
result names a recent winner. On the tests it catches 41 of 50 trick questions (see Limitations).
Files
Table with columns: file, size, what| file | size | what |
|---|
model.safetensors + config.json, generation_config.json | 248.1 MB | fp32 weights for transformers (Qwen3ForCausalLM) |
tokenizer.json, tokenizer_config.json, special_tokens_map.json | 0.55 MB | byte-level BPE, 8,192 tokens |
gguf/tiny-search-reader-60M-Q8_0.gguf |
How to run
pip install torch transformers tokenizers huggingface_hub
import sysfrom huggingface_hub import snapshot_download path = snapshot_download("ahkamboh/tiny-search-reader-60M")sys.path.insert(0, path)from searchreader import answer_transformers results = [ {"title": "Harbor Street Chess Open 2025 results", "text": "Mara Lindqvist won the 2025 Harbor Street Chess Open with 7 points from 9 games."}, {"title": "Harbor Street Chess Open 2026", "text": "The 2026 Harbor Street Chess Open was won by Tomas Varga after a tie-break against Mara Lindqvist."},]print(answer_transformers("Who won the 2026 Harbor Street Chess Open?", results)) # Tomas Varga
Without the helper, the prompt is plain text with special tokens (format below):
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "ahkamboh/tiny-search-reader-60M"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo) prompt = ("<|question|> Who won the 2026 Harbor Street Chess Open?<|snippet|>" " [1] Harbor Street Chess Open 2025 results: Mara Lindqvist won the 2025 Harbor Street Chess Open" " with 7 points from 9 games." " [2] Harbor Street Chess Open 2026: The 2026 Harbor Street Chess Open was won by Tomas Varga after a" " tie-break against Mara Lindqvist.<|answer|>")enc = tok(prompt, return_tensors="pt", add_special_tokens=False)out = model.generate(**enc, max_new_tokens=24, do_sample=False)print(tok.decode(out[0, enc.input_ids.shape[1]:], skip_special_tokens=True).strip()) # Tomas Varga
This works as long as the prompt fits. For real search results use build_prompt, which fits them into the 512-token
context the same way the model was tested.
llama.cpp
llama-completion -m gguf/tiny-search-reader-60M-Q8_0.gguf -c 512 -n 24 --temp 0 -no-cnv --no-display-prompt \ -p '<|question|> Who won the 2026 Harbor Street Chess Open?<|snippet|> [1] Harbor Street Chess Open 2025 results: Mara Lindqvist won the 2025 Harbor Street Chess Open with 7 points from 9 games. [2] Harbor Street Chess Open 2026: The 2026 Harbor Street Chess Open was won by Tomas Varga after a tie-break against Mara Lindqvist.<|answer|>'# Tomas Varga [end of text]
Tested with Homebrew llama.cpp build 11146. The GGUF says add_bos_token = false and its end-of-text token is id
0, so llama.cpp adds nothing in front and stops after the answer.
llama-cpp-python
pip install llama-cpp-python tokenizers huggingface_hub
from searchreader import answer_gguf # with path on sys.path as above print(answer_gguf("Who won the 2026 Harbor Street Chess Open?", results, gguf_path=path + "/gguf/tiny-search-reader-60M-Q8_0.gguf")) # Tomas Varga
answer_gguf builds the prompt with our own tokenizer and feeds the token ids to llama.cpp, so the special tokens are
single ids and no BOS is added. It decodes greedily and stops at <|endoftext|> (id 0) or after 24 tokens. Run
python searchreader.py for the two-question demo on both backends. Tested with transformers 5.18, llama-cpp-python
0.3.36 and tokenizers 0.23.
<|question|> {question}<|snippet|> [1] {title}: {text} [2] {title}: {text} ...<|answer|>
The model writes the answer after <|answer|> and ends it with <|endoftext|>.
Table with columns: token, id, role| token | id | role |
|---|
<|endoftext|> | 0 | end of the answer |
<|question|> | 1 | starts the question |
<|snippet|> | 2 | starts the results |
<|answer|> | 3 | the model answers after this |
|
Table with columns: rule, detail| rule | detail |
|---|
| spacing | one space before the question and before each [n], none before a special token |
| cleaning | every run of whitespace (non-breaking spaces too) becomes one space; ends are trimmed |
| one result | [n] title: text, or [n] text when there is no title; n counts from 1 in the search engine's order |
| dates | keep the date Google puts at the start of a result's text; the model uses it to tell new results from old |
| length | prompt plus answer must fit 512 tokens. Keep 24 tokens for the answer, so the prompt can use 488 |
| too long | drop results from the end until it fits; if the first result alone is too long, cut it short. A question too long to leave room for any result (over 485 tokens with the 24-token reserve) is cut to its first 256 tokens |
Five Google results usually fit: only 3 of the 258 test prompts had to drop a result.
searchreader.build_prompt(tokenizer, question, results, ctx=512, reserve=24) does all of this. It gives the same
token ids as the training code on all 258 test questions, and it takes either a tokenizers.Tokenizer or a
transformers tokenizer.
Note for apps that build the prompt as text
The special tokens must reach the model as the single ids 1, 2 and 3. If you tokenize the prompt text with llama.cpp,
turn on special-token parsing and turn off BOS:
Table with columns: API, call| API | call |
|---|
| llama.cpp C API | llama_tokenize(vocab, text, len, tokens, n_max, /*add_special*/ false, /*parse_special*/ true) |
| llama-cpp-python | llm.tokenize(text.encode(), add_bos=False, special=True) |
With these settings the GGUF's tokenizer gives the same ids as tokenizer.json on all 258 test prompts. Without
parse_special, the markers turn into ordinary text (149 tokens instead of 131 for the library demo) and in our test
the model then answered with nothing. Since the markers are parsed anywhere in the text, remove the five marker
strings from questions and results if they could ever contain them.
Results
Full tables, the test questions and the grading rules are in eval/results.md.
v3.1a (this release), PyTorch fp32 on CPU:
Table with columns: question set, questions, accuracy| question set | questions | accuracy |
|---|
| google25 (dev set, see caveats) | 24 | 91.7% (22) |
| google_heldout | 53 | 81.1% (43) |
| google_fresh | 19 | 89.5% (17) |
| google_big (collected 3-4 Oct 2026) | 162 | 76.5% (124) |
| all | 258 | 79.8% (206) |
Accuracy counts right answers plus trick questions answered with not found. On the 208 questions that have an
answer it is right 165 times, says not found 11 times and gives a wrong answer 32 times. It catches 41 of the 50
trick questions.
google_big by kind:
Table with columns: kind, example, CPU fp32, Colab bf16| kind | example | CPU fp32 | Colab bf16 |
|---|
| fact | What is the capital of Turkey? | 30/34 | 30/34 |
| live value | What is the Saudi riyal rate in Pakistan today? | 29/35 | 29/35 |
trick (right reply is not found) | When did Pakistan win the FIFA World Cup? | 24/28 | 24/28 |
| unit | What is the boiling point of mercury in Celsius? |
Earlier versions from this project, same tests (Colab GPU runs, bf16):
Table with columns: model, all 258, google_big, google_big "latest / current / most recent" questions (60)| model | all 258 | google_big | google_big "latest / current / most recent" questions (60) |
|---|
| v1.1a | 69.4% (179) | | |
| v2a | 73.3% (189) | 69.1% | 41.7% |
| v3a | 77.1% (199) | 74.7% | 65.0% |
| v3.1a | 79.8% (206) | 75.9% | 61.7% |
Paired tests (McNemar): against v2a, v3.1a fixes 17 google_big questions and breaks 6 (p=0.035), and on the "latest"
questions fixes 13 and breaks 1 (p=0.002). Against v3a it fixes 22 and breaks 15 of the 258 (p=0.32, CPU fp32), so
that gain is within noise.
Other tests (Colab bf16): SQuAD 2.0 dev F1 65.19; held-out MS MARCO and TriviaQA validation rows 63.5% accuracy.
Both were part of the score used to pick v3.1a.
Caveats:
- The test sets are small. One question moves the 258 total by 0.4 points and google_big by 0.6.
- The model was picked using Colab GPU runs in bf16. CPU fp32 also gets 206/258, but not on exactly the same
questions: on google_big it gets 124 against Colab's 123 (unit questions 21/25 against 20/25). Both are shown and
labelled.
- google25 (24 questions) is a dev set. It was the reference for the synthetic table layouts and is part of the
score used to pick v3.1a, as are real_dev and SQuAD 2.0 dev. Three google25 rows and five google_heldout rows
(latest iPhone, Galaxy S and iOS, Ballon d'Or, LPG, policy rate, dollar and petrol rates) guided the v3 data
design. On the 234 questions outside google25, v3.1a gets 184 (78.6%, CPU fp32).
- Not fully blind: v3.1 added training rows of three kinds (answer_type, traps_hard, units) after looking at v3a's
mistakes on google_fresh and google_big, so google_fresh is not fully blind for v3.1a. New v3.1 rows on google_big
subjects, or asking a test question in other words, were dropped, but that filter covered only the rows added in
v3.1. The older v1 and v1.1 synthetic rows, which v2a, v3a and v3.1a all trained on, came from fact tables that
include some google_big subjects and answers (for example Nanga Parbat's height and body temperature in
Fahrenheit). So google_big, and its unit questions most of all, is not fully blind for any of these models. For
v3.1a, only google_big questions asked word for word were removed from the older rows.
- For live values the gold list holds every value the saved results showed, since sources disagree, and any of them
counts as right.
- About a quarter of the test questions (65 of 258) are about Pakistan.
- Nothing was measured on a phone.
Compared with other models
Same 258 questions and the same grading, all on Colab GPU in bf16 with greedy decoding.
- Zero-shot models got the question and the numbered results in their own chat template (Qwen3 with thinking off),
with one instruction: "Answer the question using only the numbered search results below. Reply with just the
answer in a few words. If the results do not answer the question, reply exactly: not found".
- The two extractive models (deepset's SQuAD 2.0 readers) pick an answer span from the five results.
- SmolLM2-135M (fine-tuned) is the base SmolLM2-135M trained for 2 epochs on exactly the rows this model was
trained on.
The last column compares each model with this one question by question: questions only that model got right
against questions only this model got right, with the exact McNemar test (p below 0.05 means a real difference).
Table with columns: model, how it was run, parameters, all 258, trick questions caught (of 50), "latest" questions (of 60), only it right vs only this model right| model | how it was run | parameters | all 258 | trick questions caught (of 50) | "latest" questions (of 60) | only it right vs only this model right |
|---|
| Qwen3-4B | zero-shot | 4.02B | 91.5% (236) | 44 | 52 | 42 vs 12, p<0.001 |
| Qwen3-1.7B | zero-shot | 1.72B | 84.5% (218) | 26 |
What the table shows:
- It is clearly better than both SQuAD 2.0 readers and the three instruct models below 0.6B. It ties Qwen3-0.6B
(37 questions against 36). Qwen3-1.7B gets 12 more right, which is not a significant gap on 258 questions.
Qwen3-4B, 65 times larger, is clearly better.
- It catches 41 of the 50 trick questions, more than every model here except Qwen3-4B (44). The three instruct
models below 0.6B never replied
not found.
- It is weak on the 60 "latest / current / most recent" questions: 37 right, fewer than every generative model
tested. Qwen3-1.7B (54) and Qwen3-4B (52) are significantly better there (p of 0.001 or less).
- SmolLM2-135M fine-tuned on the same rows scores 214 against 206. That gap is not significant (p=0.358), but on the
latest questions it is (47 against 37, p=0.021), and it catches fewer trick questions (32 against 41). It has 2.2
times the parameters, and its base was pretrained on about 2 trillion tokens, against 10 billion here. So most of
the result comes from the training data, and a base model with more pretraining makes better use of it.
Caveats for this comparison:
- The zero-shot models were pretrained on the open web and may know some answers without reading the results.
This model's training held out every test question and topic.
- One fixed prompt was used for every zero-shot model. Another prompt could raise or lower their scores.
- The Qwen and SmolLM2 model cards suggest sampling; greedy decoding was used for every model so the run is
repeatable.
- google25 is this model's dev set, and google_fresh and google_big guided the v3.1 data (see the caveats above),
which favours this model a little.
Per-set scores, google_big by kind and every paired test are in eval/results.md.
Phone files
Table with columns: file, size, all 258 (llama.cpp, CPU), answers same as fp32, 449-token prompt + 24 tokens, average test question, peak RSS| file | size | all 258 (llama.cpp, CPU) | answers same as fp32 | 449-token prompt + 24 tokens | average test question | peak RSS |
|---|
tiny-search-reader-60M-Q8_0.gguf | 70.2 MB | 79.8% (206) | 254/258, all 258 verdicts the same | 317 ms | 168 ms | 174 MB |
tiny-search-reader-60M-Q4_K_M.gguf | 41.0 MB | 79.5% (205) |
Speeds are from an M1 Pro, CPU only, 4 threads, Homebrew llama.cpp. With 1 thread the 449 + 24 token case takes
982 ms (Q8_0) and 1,142 ms (Q4_K_M). On this CPU Q4_K_M is smaller but not faster. A mid-range phone should take
roughly 0.5 to 0.9 s per question; that is an estimate, not a measurement. Q8_0 is the one to use unless the 29 MB
difference matters: it gave the same answers under Homebrew llama.cpp and llama-cpp-python 0.3.36, while Q4_K_M
changed 4 of 258 answers between the two.
Speed in tokens per second
Measured with llama.cpp on an Apple M1 Pro, CPU only (no GPU, no Metal), on a real 449-token test prompt plus a
24-token answer, median of 5 runs. "Reading" is the prompt pass, "writing" is generating the answer.
Table with columns: file, threads, reading (tokens/s), writing (tokens/s), 449-token prompt + 24-token answer| file | threads | reading (tokens/s) | writing (tokens/s) | 449-token prompt + 24-token answer |
|---|
tiny-search-reader-60M-Q8_0.gguf | 1 | 558 | 135 | 982 ms |
tiny-search-reader-60M-Q8_0.gguf | 2 | 1,100 | 210 | 522 ms |
tiny-search-reader-60M-Q8_0.gguf |
A typical test question has about 317 prompt tokens and a 5-token answer, so most of the time goes into reading the
results: 168 ms per question with Q8_0 at 4 threads. These numbers come from Homebrew's prebuilt llama.cpp; in one
check a llama.cpp built on this machine was about twice as fast, so treat them as a floor for this Mac.
Compared with Qwen3-0.6B
Same Mac, same llama.cpp build (Homebrew build 11146), llama-bench -p 449 -n 24 -r 5 -ngl 0 (all layers on the CPU),
both files Q8_0. llama-bench measures writing from an empty context, so its writing numbers are higher than in the
table above, where the answer is written after the 449-token prompt.
Table with columns: model, file, parameters, reading (tokens/s), 1 / 4 threads, writing (tokens/s), 1 / 4 threads, 449-token prompt + 24-token answer, 4 threads| model | file | parameters | reading (tokens/s), 1 / 4 threads | writing (tokens/s), 1 / 4 threads | 449-token prompt + 24-token answer, 4 threads |
|---|
| tiny-search-reader-60M | 70 MB | 62M | 575 / 1,900 | 310 / 561 | 0.28 s |
| Qwen3-0.6B | 634 MB | 596M | 89 / 340 | 82 / 132 | 1.50 s |
On this machine it answers about 5 times faster than Qwen3-0.6B and its file is 9 times smaller. Accuracy against
Qwen3-0.6B is in Compared with other models.
Training
Architecture
Table | |
|---|
| layout | decoder-only transformer, loads as Qwen3 (Qwen3ForCausalLM) |
| parameters | 62,018,304, input and output embeddings tied |
| layers, width | 18, 512 |
| attention | 8 heads, 8 KV heads, head dim 64, per-head QK-norm, RoPE theta 10,000 |
| MLP | SwiGLU, hidden size 1,408 |
| norm | RMSNorm, eps 1e-5 |
| context | 512 tokens |
| tokenizer | byte-level BPE, 8,192 tokens including the 5 special tokens |
Pretraining
Table | |
|---|
| data | 10B tokens of FineWeb-Edu (sample-10BT) |
| optimizer | Muon (weight decay 0.01) for the hidden weight matrices, AdamW for the rest |
| precision | bf16 |
| hardware | one RTX PRO 6000 on Colab, about 10.6 hours |
| final validation loss | 2.7127 |
| search reading | search-reading examples mixed into the last 30% of training, while the learning rate decays |
Fine-tuning
2 epochs on real and synthetic search-reading rows. Every training row that asks a test question word for word was
dropped, and so was every row on a google25, google_heldout or google_fresh topic. The rows added in v3.1 also left
out google_big subjects and test questions asked in other words (see the caveats under Results).
Table with columns: source, rows| source | rows |
|---|
| MS MARCO v2.1 | real Bing queries with up to 5 of their web passages; "No Answer Present" becomes not found |
| TriviaQA (unfiltered) | trivia questions with Bing's top 5 results (title and description) |
| SQuAD 2.0 | questions on one Wikipedia paragraph (train split) |
| hard negatives | copies of answerable MS MARCO and TriviaQA rows where every result holding the answer is swapped for on-topic results without it; the reply is not found |
| synthetic, Google-style | written by Qwen/Qwen3.6-35B-A3B-FP8 on vLLM. Python picks the facts that decide each answer and the LLM writes result text around them. A second, temperature-0 pass of the same LLM answers from the results alone, and a row is kept only when it agrees with Python's answer and passes Python's checks |
Synthetic categories:
Table with columns: category, teaches, added in| category | teaches | added in |
|---|
| list_numbers | read one cell of a list or table (rates, prices, scores, prayer times) | v1 |
| recency | several dated results with different values; the newest wins | v1 |
| answer_type | the result also holds a date, the wrong unit or another number | v1, more rows in v3.1 |
| traps | future, false-premise, fictional or wrong-entity questions; the reply is not found | v1 |
| multi | plain facts; one result has the answer, the others are near misses, ads and junk |
Limitations
- English only.
- It answers from the results you give it. It does not search and is not meant to answer from memory, so poor
results give poor answers.
- About 20% of answers on the tests are wrong, and a wrong answer is more common than a
not found. Show the
source result next to the answer.
not found works best on false-premise and future questions (41 of 50 caught). It is weaker when the results are
about the right subject but leave out the asked fact. In a hand check of 6 made-up cases like that, it said
not found once and the other five times gave a nearby time or number, such as 8 for "How many books does the
Harbor Street library have?" from a result saying the library is open from 8 am to 8 pm.- "Latest" and "current" answers depend on the dates in the results. The model does not know today's date; it picks
the result that looks newest. Latest-version (6/14) and most-recent-winner (14/26) questions are its weakest kinds, and every other
generative model tested did better on the 60 "latest" questions.
- 512 tokens of context, about five Google-length results.
- No chat and no instructions: one question in, a few words out, in the prompt format above only.
License and data notes
The weights, searchreader.py and eval/grade.py are under Apache-2.0 (see LICENSE).
Table with columns: data, used for, terms| data | used for | terms |
|---|
| FineWeb-Edu (sample-10BT) | pretraining | ODC-By 1.0 |
| MS MARCO v2.1 | fine-tuning | non-commercial research terms |
| TriviaQA | fine-tuning | the dataset's own terms |
| SQuAD 2.0 | fine-tuning | CC BY-SA 4.0 |
| Qwen3.6-35B-A3B outputs | synthetic fine-tuning rows | the model is Apache-2.0 |
| Google search results |
MS MARCO's terms are for non-commercial research. If you plan commercial use, check whether that matters for you.
Author
Ali Hamza Kamboh (GitHub: ahkamboh, Hugging Face: ahkamboh).