How to use
import torchfrom huggingface_hub import hf_hub_downloadfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "Kausp11/querylm-annotator-gemma4-e4b-lora"tok = AutoTokenizer.from_pretrained(repo)base = AutoModelForCausalLM.from_pretrained("google/gemma-4-E4B-it", torch_dtype=torch.bfloat16)model = PeftModel.from_pretrained(base, repo).eval() # The instruction the adapter was trained with. Its <WEB_*> tag names are mapped to Gemma's native tool tokens.instruction = open(hf_hub_download(repo, "annotator_instruction.txt")).read().strip()for src, dst in (("<WEB_QUERY>", "<|tool_call>"), ("</WEB_QUERY>", "<tool_call|>"), ("<WEB_RESPONSE>", "<|tool_response>"), ("</WEB_RESPONSE>", "<tool_response|>")): instruction = instruction.replace(src, dst)prefix = tok.apply_chat_template([{"role": "user", "content": instruction}], tokenize=False, add_generation_prompt=True)Q_OPEN, Q_CLOSE = tok.convert_tokens_to_ids("<|tool_call>"), tok.convert_tokens_to_ids("<tool_call|>") document_so_far = ("Albedo is the fraction of sunlight that is diffusely reflected by a body. " "It is measured on a scale from")ids = tok(prefix + document_so_far, add_special_tokens=False, return_tensors="pt").input_idswith torch.no_grad(): # 1) How likely is a search at this position? (annotation keeps positions with probability >= 0.05) p_search = torch.softmax(model(input_ids=ids).logits[0, -1].float(), -1)[Q_OPEN].item() # 2) The query the annotator would write here (greedy, at most 24 tokens). out = model.generate(input_ids=torch.cat([ids, torch.tensor([[Q_OPEN]])], 1), max_new_tokens=24, do_sample=False, eos_token_id=Q_CLOSE)query = tok.decode(out[0, ids.shape[1] + 1:], skip_special_tokens=False).replace("<tool_call|>", "").strip()print(f"P(search here) = {p_search:.3f}")print("query:", query)
Output of this exact script (CPU, bfloat16, torch 2.8.0, peft 0.13.2, run 2026-11-20 on the files in this repo):
P(search here) = 0.076query: Albedo scale fraction sunlight reflected body range
How it was used to annotate a corpus: for every position in a document, compute the probability of the query-open token as above; keep positions with probability of at least 0.05 that are at least 40 characters apart; at each kept position generate the query greedily (at most 24 tokens). The retrieved passage is then inserted after the query by the pipeline. The adapter does not produce passages.
Tokens
The adapter uses Gemma's native tool tokens, so no token was added to the vocabulary: <|tool_call> and <tool_call|> around a query, <|tool_response> and <tool_response|> around a passage. In the QueryLM models and datasets the same four roles are played by <WEB_QUERY>, </WEB_QUERY>, <WEB_RESPONSE>, </WEB_RESPONSE>.
Training data
About 10,000 English Wikipedia articles (the first articles of the November 2023 dump with 2,000 to 9,000 characters), annotated by a larger teacher language model in two passes: it first marks where a search would help predict an upcoming fact, then writes a query for each position from the preceding text only. For each query, passages were retrieved with BM25 over FineWeb-Edu and compressed to one to three sentences. No relevance filter was applied to the queries. About 20 queries per article; 9,139 articles for training.
Training procedure
Table | |
|---|
| Objective | next-token loss on the document text, the queries, the four boundary tokens and the end-of-turn marker; the instruction and the passages carry no loss |
| Input | one annotated article per sequence, after the instruction (rendered with the chat template); truncated at 4,096 tokens |
| Optimizer | AdamW, peak LR 2e-4, 3% warmup, cosine decay |
| Batch | 32 sequences per update |
| Length | 3 epochs, 858 updates |
Provenance
Table | |
|---|
| Checkpoint | run none_10k_lora, final adapter after 3 epochs (858 of 858 updates). The final checkpoint was used, not one selected on validation loss |
adapter_model.safetensors sha256 | 09ed003ad663ea8188e037ce7bdda922aab0de68f16489897ceab4643de7b328 |
Intended use and limitations
- A research artifact: the annotator behind the QueryLM training data. Not a general assistant.
- It was trained on Wikipedia articles and then applied to web text; its query style follows the teacher's annotations.
- It decides where to search and what to ask. It does not judge whether a retrieved passage is correct or relevant; the data pipeline filters insertions afterwards.
- English only.
License
Apache-2.0. The licence text is in LICENSE.
This adapter was fine-tuned from google/gemma-4-E4B-it, which Google releases under the Apache License 2.0 (licence text). Modification notice: the files adapter_model.safetensors and adapter_config.json are new; they hold low-rank weight updates obtained by fine-tuning the base model on the data described above. The tokenizer files and chat_template.jinja are unmodified copies from the base model. This is an independent research artifact and is not affiliated with or endorsed by Google.
Citation
To be added.