Checkpoint
- Base model:
allenai/OLMoE-1B-7B-0125
- Base revision:
9b0c1aa87e34a20052389dce1f0cf01da783f654
- Dataset revision:
740312add88f781978c0658806c59bc2815b9866
- Method: full LoRA on every expert in every layer
- LoRA rank/alpha: 32/64
- Targets: attention, routers, and all 64 experts per layer
- Trainable parameters: 311,459,840 (4.5014%)
- Training: 3 epochs, effective batch size 128, learning rate 4e-4
- Seed: 42
- Weight SHA-256:
3d288c7374e93356ff7f41fb94ce27dabf688812ce75dd02bb9c728252fcc102
This is the replication project's custom adapter-only format, not a standard
PEFT checkpoint. Use the loader in the HELLoRA-replication source repository
to inject the recorded all-expert topology before loading
adapter_model.safetensors.
The saved adapter was verified by loading it over a fresh copy of the pinned
base model and checking bit-exact output-logit parity with the trained model.
Held-out GSM8K result
Under the replication's format-tolerant answer extraction:
- pretrained base: 46.47% (613/1,319);
- full LoRA: 41.17% (543/1,319);
- HELLoRA: 39.58% (522/1,319).
Full LoRA leads HELLoRA by 1.59 points, but the paired difference is not
statistically decisive (p = 0.286; paired-bootstrap 95% interval -1.21 to
+4.40 points). Under a common final-number scorer, the gap is 1.44 points and
also not decisive (p = 0.335).
The base's 46.47% is not a parser-neutral comparison: the base never emits
#### and uses an answer-phrase fallback. Strict shared-marker scoring gives
base/full-LoRA accuracies of 0.00%/41.17%; common-final-number scoring gives
1.97%/41.02%.
Training summary
- Optimizer steps: 177
- Task loss by epoch: 0.35340, 0.15313, 0.08246
- Peak allocated VRAM: 36.18 GiB
- Peak reserved VRAM: 66.18 GiB
- Training duration: 630.21 seconds
Files
adapter_model.safetensors: adapter tensors only
adapter_config.json: topology, pinned revisions, and checksum
run_config.json: training configuration
metrics.json: training and checkpoint verification metrics