Checkpoint
- Base model:
allenai/OLMoE-1B-7B-0125
- Base revision:
9b0c1aa87e34a20052389dce1f0cf01da783f654
- Dataset revision:
740312add88f781978c0658806c59bc2815b9866
- Method: HELLoRA, layer-level top-8 hot experts
- LoRA rank/alpha: 32/64
- Targets: attention, routers, selected experts
- Trainable parameters: 47,218,688 (0.6824%)
- Seed: 42
- Weight SHA-256:
07cef4b06c0022db444ef7e21bed6e6b4e9393d55aa3aa26d791cd2cc339ebf3
This is a custom HELLoRA checkpoint, not a standard PEFT checkpoint. The
selective expert topology must be reconstructed from hot_experts.json before
loading adapter_model.safetensors. Use the loader in the
HELLoRA-replication source repository.
Held-out GSM8K result
Under the replication's format-tolerant answer extraction:
- pretrained base: 46.47% (613/1,319);
- HELLoRA: 39.58% (522/1,319).
This is not a parser-neutral base comparison. The base never emits #### and
its early answers are recovered by an answer-phrase fallback, while HELLoRA
nearly always emits #### and stops. A strict shared marker regex scores the
base at 0.00% and HELLoRA at 39.58%; a shared final-number regex scores them at
1.97% and 39.58%, respectively.
The adapter did learn the requested output format: 99.55% of generations used
the #### marker and stopped at EOS.
Files
adapter_model.safetensors: adapter tensors only
adapter_config.json: topology, revisions, and checksum
hot_experts.json: selected experts for every MoE layer
run_config.json: training configuration
metrics.json: training and checkpoint verification metrics