GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
icedduck
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
icedduck
Model Tree
allenai/Olmo-3-7B-Think-SFT
Input Modalities
Output Modalities
Supported Functionality
One of three small DPO runs trained to test where Olmo 3's verbosity comes from (allenai/open-instruct issue #2002). All three start from allenai/Olmo-3-7B-Think-SFT, use Ai2's 7B Think DPO recipe and the same seed, and see 20,000 preference pairs each; the only difference between them is which pairs. This checkpoint is the dose-matched baseline: what Ai2's own pairs do at 13% of the data under the same recipe.
Pairs. 20,000 pairs drawn at random from allenai/Dolci-Think-DPO-7B (the pairs as Ai2 released them). Chosen answers are longer than rejected ones in 83.9% of pairs, median chosen/rejected answer-length ratio 1.80 (523 / 253 median tokens). The exact pair ids are in the GitHub repo below (experiments/2026-10-22_dpo-feasibility/arms/), so the set can be rebuilt from the public dataset.
Recipe (open-instruct dpo_tune_cache.py, commit ff32794a3, Olmo 3 report Table 48 settings): length-normalised DPO (dpo_norm), β 5, learning rate 8e-8 with linear decay, 1 epoch, max sequence length 16,384, chat template olmo_thinker (the Think template without the released model's default system prompt), weight decay 0, bf16, flash attention. Deviations from Ai2's full run: 20,000 pairs instead of 150,000 (13%), effective batch 30 (1 × 3 GPUs × 10 accumulation steps) instead of 128, no warm-up instead of 0.1. Trained on 3 × H100 80 GB with the optimizer state offloaded to CPU, 667 steps, final logged loss about 0.50.
What it does (every model evaluated the same way: own chat template, temperature 0.6, top_p 0.95, one sample per prompt, 32K token cap; answer = text after the last </think>; the control is Think-SFT under the same training template; bootstrap 95% intervals):
| this arm | control Think-SFT | Ai2 Think-DPO | |
|---|---|---|---|
| AlpacaEval 805 prompts, median answer tokens | 647 | 577 | 874 |
| paired change vs control (median) | +35 [+25, +46] | – | +243 [+231, +258] |
| prompts with a longer answer than the control | 62% | – | 94% |
| IFEval prompt-strict (541) | 0.756 | 0.784 | 0.719 |
| GSM8K (1,319) | 0.939 | 0.946 | 0.908 |
answers with \boxed{} on GSM8K | 1,214 | 1,260 | 625 |
| held-out set, median answer tokens, short / detail / open | 139 / 821 / 750 (A lengthens detail answers by +58 [+12, +88] and open answers by +28 [+14, +66] tokens over the control) | 138 / 744 / 729 | 295 / 1048 / 966 |
Arm-vs-arm on the same 805 prompts: –.
Reading (with the caveats below): the chosen-vs-rejected length asymmetry of the pairs is a main cause of the verbosity, carrying more than half of the length shift at this dose; removing it costs nothing measurable on IFEval, GSM8K or formatting. Full tables, judge records and scripts: https://github.com/bethehand/olmo-rlvr-side-effects (experiments/2026-10-23_dpo-arms-eval/).
Caveats. One training seed per arm (B′ is a second selection, not a second seed); 13% of Ai2's pair count, batch 30, no warm-up; 7B only, so the result says direction, not magnitude at full scale; a Qwen3-32B pairwise judge ties A vs B and slightly favours A over B′, but picks the longer answer in 79% of decided arm-vs-arm prompts, so quality is "not clearly worse", not "proven equal".
Loading. The checkpoint was saved by transformers 5.4. config.json carries both the 5.x rope_parameters block and the 4.x keys (rope_theta 500000, YaRN rope_scaling, factor 8) so that transformers 4.57 / vLLM 0.11 load the RoPE correctly; without the 4.x keys, 4.57 silently fell back to θ = 10000 and the model looked broken. The loaded config and tokenizer were checked against the start model's. chat_template.jinja is the training template.
python
from transformers import AutoModelForCausalLM, AutoTokenizerm = AutoModelForCausalLM.from_pretrained("icedduck/olmo3-7b-think-dpo-arm_A_seed1", torch_dtype="auto")t = AutoTokenizer.from_pretrained("icedduck/olmo3-7b-think-dpo-arm_A_seed1")from transformers import AutoModelForCausalLM, AutoTokenizerm = AutoModelForCausalLM.from_pretrained("icedduck/olmo3-7b-think-dpo-arm_A_seed1", torch_dtype="auto")t = AutoTokenizer.from_pretrained("icedduck/olmo3-7b-think-dpo-arm_A_seed1")