Model and training
- Base:
Qwen/Qwen2.5-14B-Instruct, revision cf98f3b3bbb457ad9e2bb7baf9a0125b6b88caa8.
- Parameters: 14,770,033,664; BF16; original 32,768-token native context configuration.
- Source data: Xirui1208/readall-sft-stage-a-b, revision
877b52deb0622bed8f8102693f6907413df7d693.
- Local quality revision: 73,400 training rows and 1,575 validation rows; 16 training rows were quarantined and 101 benchmark-overlap validation rows were removed.
- Global batch 256; learning rate 1e-5; warmup ratio 0.1; four GPUs; max training sequence length 12,288. Detailed settings and accumulation adaptation are in
training_metadata.json.
Loading
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = "Xirui1208/readall-readtwice-14b-stage-a-step230-20260923"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto")
The checkpoint supports the recurrent ReadAll / ReadTwice protocol: first SKIM every source chunk to construct an outline, then revisit every original chunk with the question, outline and evolving memory, and finally answer from memory. KEEP_MEMORY retains the preceding memory; other UPDATE text replaces it. Memory rewriting follows the original Stage A protocol.
The long-document results below require this external recurrent and paginated evaluation loop. They do not indicate a native 112K context window or a single generate call on the entire document.
Recorded evaluation
Table with columns: Pinned HQA length, Questions, SubEM correct, SubEM, EM, Token F1| Pinned HQA length | Questions | SubEM correct | SubEM | EM | Token F1 |
|---|
| 56K | 128 | 96 | 75.00% | 60.16% | 71.99% |
| 112K | 128 | 99 | 77.34% | 60.94% | 74.38% |
These are the existing completed evaluations of this exact Stage A step230 export. The same 128 questions are reused at both lengths. No completed 224K or LongBench score is available for this checkpoint.
Protocol: temperature 0, top_p 1, seed 801; all chunks read in both passes; lossless chunks of at most 5,000 tokens; SKIM/UPDATE/FINAL output budgets 512/1,024/1,024. Outlines are paginated with at most 24 sections per page and an 8,192-token navigation prompt budget; memory is carried serially between pages. Gold labels are used only for scoring. SubEM is normalized bidirectional substring matching, with strict yes/no/noanswer handling; it is distinct from exact match.
The tokenizer, model configuration and saved generation configuration are preserved from the export. The saved generation defaults differ from the benchmark settings, so set benchmark sampling options explicitly. See evaluation_protocol.json and evaluation_results.json.
Contents
Six safetensors weight shards, their index, tokenizer assets, model/generation configuration, training metadata and evaluation summaries. The original distributed optimizer checkpoint remains on the training machine and is not part of this inference release.