Recorded HQA evaluation
Table with columns: Length, Questions, SubEM correct, SubEM, EM correct, EM| Length | Questions | SubEM correct | SubEM | EM correct | EM |
|---|
| 56K | 128 | 100 | 78.125% | 80 | 62.500% |
| 112K | 128 | 101 | 78.90625% | 78 | 60.9375% |
| 224K | 128 | 92 | 71.875% | 71 | 55.46875% |
All samples completed successfully. The same 128 question IDs are reused at each length. Scores use the historical MemAgent normalization and bidirectional substring SubEM rule, with no manual corrections. Exact metrics and input hashes are in evaluation.json.
Evaluation uses the external recurrent, paginated ReadTwice loop: SKIM all source chunks in a page to construct a page-local outline, read every original chunk again while updating memory, then answer from the final memory. Memory carries across pages; <KEEP_MEMORY> retains the existing memory. Parameters: lossless chunks of at most 5,000 tokens, at most 24 sections per page, SKIM/UPDATE/FINAL limits 512/1024/1024, temperature 0, top_p 1, seed 801, and unguarded MemAgent-compatible memory updates. Evaluation used DP4 x TP2 on eight GPUs.
The native model context remains 32,768 tokens. These long-document scores require the recurrent evaluation loop.
Download and load
This repository is public. Downloads do not require authentication.
from huggingface_hub import snapshot_downloadsnapshot_download('Xirui1208/readall-readtwice-14b-rl-step15-20260925', local_dir='readtwice_14b_step15')
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model_id = 'Xirui1208/readall-readtwice-14b-rl-step15-20260925'tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map='auto')
Set evaluation sampling parameters explicitly. The saved generation configuration is preserved from the SFT export.