GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
icedduck
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
icedduck
Model Tree
allenai/Olmo-3-7B-Think-SFT
Input Modalities
Output Modalities
Supported Functionality
One of three small DPO runs trained to test where Olmo 3's verbosity comes from (allenai/open-instruct issue #2002). All three start from allenai/Olmo-3-7B-Think-SFT, use Ai2's 7B Think DPO recipe and the same seed, and see 20,000 preference pairs each; the only difference between them is which pairs. This checkpoint is the intervention: the same recipe and data size without the systematic 'chosen is longer' pattern.
Pairs. 20,000 pairs with chosen/rejected answer-length ratio ≤ 1.5, sampled to match arm A's chosen-length decile distribution. Chosen longer in 65.0% of pairs, median ratio 1.13 (523 / 473 median tokens). Its source mix drifted (more math than A); arm B′ fixes that. The exact pair ids are in the GitHub repo below (experiments/2026-10-22_dpo-feasibility/arms/), so the set can be rebuilt from the public dataset.
Recipe (open-instruct dpo_tune_cache.py, commit ff32794a3, Olmo 3 report Table 48 settings): length-normalised DPO (dpo_norm), β 5, learning rate 8e-8 with linear decay, 1 epoch, max sequence length 16,384, chat template olmo_thinker (the Think template without the released model's default system prompt), weight decay 0, bf16, flash attention. Deviations from Ai2's full run: 20,000 pairs instead of 150,000 (13%), effective batch 30 (1 × 3 GPUs × 10 accumulation steps) instead of 128, no warm-up instead of 0.1. Trained on 3 × H100 80 GB with the optimizer state offloaded to CPU, 667 steps, final logged loss about 0.56.
What it does (every model evaluated the same way: own chat template, temperature 0.6, top_p 0.95, one sample per prompt, 32K token cap; answer = text after the last </think>; the control is Think-SFT under the same training template; bootstrap 95% intervals):
| this arm | control Think-SFT | Ai2 Think-DPO | |
|---|---|---|---|
| AlpacaEval 805 prompts, median answer tokens | 613 | 577 | 874 |
| paired change vs control (median) | +15 [+6, +22] | – | +243 [+231, +258] |
| prompts with a longer answer than the control | 56% | – | 94% |
| IFEval prompt-strict (541) | 0.765 | 0.784 | 0.719 |
| GSM8K (1,319) | 0.947 | 0.946 | 0.908 |
answers with \boxed{} on GSM8K | 1,255 | 1,260 | 625 |
| held-out set, median answer tokens, short / detail / open | 150 / 774 / 770 | 138 / 744 / 729 | 295 / 1048 / 966 |
Arm-vs-arm on the same 805 prompts: B − A = −19 [−30, −7] tokens (B shorter on 451 of 805 prompts, longer on 347; second generation seed −14 [−25, −6]).
Reading (with the caveats below): the chosen-vs-rejected length asymmetry of the pairs is a main cause of the verbosity, carrying more than half of the length shift at this dose; removing it costs nothing measurable on IFEval, GSM8K or formatting. Full tables, judge records and scripts: https://github.com/bethehand/olmo-rlvr-side-effects (experiments/2026-10-23_dpo-arms-eval/).
Caveats. One training seed per arm (B′ is a second selection, not a second seed); 13% of Ai2's pair count, batch 30, no warm-up; 7B only, so the result says direction, not magnitude at full scale; a Qwen3-32B pairwise judge ties A vs B and slightly favours A over B′, but picks the longer answer in 79% of decided arm-vs-arm prompts, so quality is "not clearly worse", not "proven equal".
Loading. The checkpoint was saved by transformers 5.4. config.json carries both the 5.x rope_parameters block and the 4.x keys (rope_theta 500000, YaRN rope_scaling, factor 8) so that transformers 4.57 / vLLM 0.11 load the RoPE correctly; without the 4.x keys, 4.57 silently fell back to θ = 10000 and the model looked broken. The loaded config and tokenizer were checked against the start model's. chat_template.jinja is the training template.
python
from transformers import AutoModelForCausalLM, AutoTokenizerm = AutoModelForCausalLM.from_pretrained("icedduck/olmo3-7b-think-dpo-arm_B_seed1", torch_dtype="auto")t = AutoTokenizer.from_pretrained("icedduck/olmo3-7b-think-dpo-arm_B_seed1")from transformers import AutoModelForCausalLM, AutoTokenizerm = AutoModelForCausalLM.from_pretrained("icedduck/olmo3-7b-think-dpo-arm_B_seed1", torch_dtype="auto")t = AutoTokenizer.from_pretrained("icedduck/olmo3-7b-think-dpo-arm_B_seed1")