lixiaochuan2020

qwen3-1.7b-midtrain-local-kv-per-document

Deploy Dedicated

README

License: apache-2.0

Access

Public: anyone can download it, no access request or login needed.

Weights

Saved dtype: float32 (as written by training). Load with dtype=torch.float32, or cast to bfloat16 for inference.
Branch main: the final checkpoint, step 4,763 of 4,763. Intermediate checkpoints, if added later, go on branches named step-<N>.

Mid-training data

A Qwen/Qwen3-32B teacher (thinking mode) answered NQ and HotpotQA training questions as a search agent, 4 times per question, using a wiki_search tool over wiki-18 (e5-base-v2, 5 passages per search, at most 10 searches, at most 16,384 tokens per trajectory). Trajectories were kept if they called at least one search and the Qwen3-32B answer judge marked the final answer correct. Gold answers were used only for that check, never as training text.

Table with columns: Item, Value
Item	Value
Questions answered	128,875
Trajectories kept	297,720
Supervised (assistant) tokens	243.78M
Tool-result (passage) tokens	314.24M
Total tokens	684.86M

The loss covers assistant tokens only.

Mid-training config

Table with columns: Setting, Value
Setting	Value
Base model	`Qwen/Qwen3-1.7B`
Optimizer	Muon for the decoder layers' weight matrices, AdamW for embeddings and norms
Learning rate	2e-5
Schedule	cosine, 3% warmup
Epochs	1
Steps	4,763
Packed row length (tokens)	18,432 (padding-free; each trajectory attends only to itself)
Rows per step	8 (8 GPUs × 1 row)
Weight decay	0
Max gradient norm	1.0
Precision	bf16 autocast, float32 master weights

Attention

Each retrieved passage, from its Doc k(Title: token through its final newline, is its own document. Its tokens attend only to earlier tokens of the same passage: not to the conversation before the search, and not to the other passages.
Passage K/V are detached: the loss on later tokens does not train how passages are encoded.
The tool-turn wrapper tokens (<|im_start|>user\n<tool_response>\n before the passages and </tool_response><|im_end|>\n after them) attend normally.
Every token after the tool turn attends to all passages.
Passages keep their absolute positions in the sequence.

Prompt format

Qwen3 chat template with one tool, wiki_search, called in Qwen3's <tool_call> format. Each tool result lists the top 5 wiki-18 passages as Doc k(Title: …) text lines. The system prompt and tool schema used in training are SYSTEM_PROMPT and TOOLS in src/kv_midtrain/data_loading/afm_mhqa.py of ashmitkx/kv-midtrain (private repository).

Evaluation

Only measured numbers; "pending" where an eval has not been run.

Table with columns: Eval, Setting, Metric, Value
Eval	Setting	Metric	Value
Held-out search QA: 1,200 questions (720 HotpotQA dev, 240 NQ test, 240 2WikiMultihopQA dev), 3 samples each	self-generate with live wiki-18 search, temperature 0.7, top-p 0.95, per-document local-KV	Judge accuracy (Qwen3-32B, final answer)	pending
GSM8K test (1,319 problems)	no tools, 1 sample, temperature 0.6, top-p 0.95, top-k 20	Judge accuracy (Qwen3-32B, final `\boxed{}` answer)	pending
MATH-500 (500 problems)	same	same	pending
Out-of-distribution search: Bamboogle, MuSiQue, PopQA, FEVER (1,025 questions)	self-generate with live wiki-18 search, temperature 0.7, top-p 0.95, per-document local-KV	Judge accuracy (Qwen3-32B, final answer)	pending

Training run

W&B: local-KV-per-document

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

lixiaochuan2020

Model Tree

Base

Qwen/Qwen3-1.7B

Fine-tuned

this model

Input Modalities

Text

Output Modalities