Config
- seq_len 4096, sample packing (concat chunks), assistant-only loss masking
- bf16 mixed precision on 1x NVIDIA H100, SDPA attention
- optimizer adamw_bnb_8bit (bf16), lr 2e-05 cosine, warmup 100, wd 0, seed 42
- effective batch 8 packed seqs (32,768 tokens/step), 1 epoch (10,445 steps)
- final train_loss 1.301
v1 parity notes
Same base/data/lr/schedule as saidutta69/RaceBench-MiniCPM5 (v1.0); deviations are hardware-forced only:
- bf16 vs fp16 (H100-native; also required by transformers 5.x)
- SDPA attention vs flash-attn
- 8-bit adam states
- single GPU with grad_accum 8 vs T4x2 grad_accum 4 (identical effective batch)
Results
lm-eval v0.4.12 results pending — see results/ folder and results-v1.1.json.
Citation
@misc{racebench2026,
title={RaceBench: Making sub-3B models replace API-based frontier models in edge deployments},
author={Sai Dutta Abhishek Dash},
year={2026},
url={https://huggingface.co/datasets/saidutta69/RaceBench-v1.1}
}