zaydsyed

Qwen2.5-0.5B-Instruct-DPO-GRPO

README

License: apache-2.0

Model details

Developed by: Zayd Syed
Base model: Qwen/Qwen2.5-0.5B-Instruct
Parameters: 494M, bf16
Language: English
License: Apache 2.0, same as the base model
Demo: runs in the browser at models.zaydsyed.com

Intended use

A small assistant for chat and text tasks where following the user's instructions closely matters, such as formatting, length and style constraints, on hardware too small for larger models, including fully in a web browser. It is a research and demonstration model.

Not intended for factual lookups, medical, legal or financial advice, or any decision with real consequences: at 0.5B parameters it is often confidently wrong.

Results

All models evaluated with the same harness: vLLM, greedy decoding, the model's own chat template, no system prompt. Generation tasks allow up to 1,280 new tokens, the default in lm-evaluation-harness's IFEval config; MMLU-Redux allows 32 since it only needs a letter.

Table with columns: Model, IFEval, IFBench, GSM8K, MMLU-Redux
Model	IFEval	IFBench	GSM8K	MMLU-Redux
Qwen2.5-0.5B-Instruct	25.5	10.0	47.0	40.3
+ DPO (Qwen2.5-0.5B-Instruct-DPO)	39.0	13.3	47.5	38.9
+ DPO + GRPO (this model)	44.4	18.7	53.3	38.2
Qwen2.5-1.5B-Instruct, for reference	41.0	15.0	75.3	62.9

Change from the base with paired bootstrap 95% confidence intervals (percentage points):

Table with columns: IFEval, IFBench, GSM8K, MMLU-Redux
	IFEval	IFBench	GSM8K	MMLU-Redux
base → DPO	+13.5 [+9.6, +17.6]	+3.3 [+0.0, +7.0]	+0.5 [−1.9, +3.0]	−1.4 [−2.3, −0.6]
DPO → GRPO	+5.4 [+1.3, +9.4]	+5.3 [+1.0, +9.7]	+5.8 [+3.2, +8.4]	−0.8 [−1.6, +0.1]
base → this model	+18.9 [+14.8, +22.9]	+8.7 [+4.0, +13.0]	+6.3 [+3.8, +8.9]	−2.1 [−3.2, −1.1]

On instruction following the 0.5B model now scores at or above the 3× larger Qwen2.5-1.5B-Instruct (IFEval +3.3 [−1.1, +8.1], IFBench +3.7 [−1.3, +8.3]; ahead on both, though neither gap is statistically significant). It remains far behind on math and knowledge, which post-training cannot add.
IFBench uses constraint types that appear in none of the training data, so its gain is evidence of generalization rather than memorized constraint types.
MMLU-Redux drops 2.1 points, the usual alignment tax from moving away from the base model.

Open-ended chat quality. Head-to-head on 300 randomly sampled AlpacaEval prompts, each pair judged twice with the answer order swapped, Claude Haiku as judge (ties count as half a win):

Table with columns: This model vs, Win rate [95% CI]
This model vs	Win rate [95% CI]
Qwen2.5-0.5B-Instruct (base)	54.2% [49.0, 59.4]
Qwen2.5-0.5B-Instruct-DPO	55.7% [50.7, 60.4]
Qwen2.5-1.5B-Instruct (125 prompts)	26.0% [19.2, 33.2]

So the instruction-following and math gains come with no loss in general answer quality against the base, while the 3× larger model still writes clearly better open-ended answers.

Metrics. IFEval: prompt-level strict accuracy, 541 prompts, scored with lm-eval-harness's IFEval checker (langdetect seeded for reproducibility). IFBench: prompt-level strict, 300 prompts, scored with AI2's official script. GSM8K: exact match on the last number, 1,319 test problems, with the instruction to end on "Answer: <number>". MMLU-Redux: the 5,330 questions labelled correct in MMLU-Redux 2.0, zero-shot, first A–D letter in the reply. This protocol scores the base model higher than Qwen's official MMLU-Redux setup (40.3 vs 24.1), so only compare within this table. Our IFEval and GSM8K numbers for the base reproduce Qwen's reported 27.9 and 49.6 within about 2.5 points.

Usage

python

from transformers import pipeline pipe = pipeline("text-generation", model="zaydsyed/Qwen2.5-0.5B-Instruct-DPO-GRPO", dtype="bfloat16")messages = [{"role": "user", "content": "Describe the moon in exactly three sentences, all in lowercase."}]print(pipe(messages, max_new_tokens=256)[0]["generated_text"][-1]["content"])

With vLLM:

python

from vllm import LLM, SamplingParams llm = LLM("zaydsyed/Qwen2.5-0.5B-Instruct-DPO-GRPO")out = llm.chat([{"role": "user", "content": "List three uses of copper as a bulleted list."}], SamplingParams(temperature=0.7, max_tokens=256))print(out[0].outputs[0].text)

Training

Both stages are full fine-tunes in bf16 on one A100 40 GB with TRL, following the DPO-then-RLVR recipe and data of Tulu 3.

Stage 1, DPO. Trained with DPO, a method introduced in Direct Preference Optimization: Your Language Model is Secretly a Reward Model. 38.3K preference pairs, half from allenai/tulu-3-pref-personas-instruction-following and half from allenai/olmo-2-0425-1b-preference-mix. Pairs whose prompt shared any 10-word sequence with IFEval, GSM8K or IFBench were dropped. β = 0.1, learning rate 5e-6 with linear decay and 10% warmup, effective batch 64, max length 2,048, one epoch (589 steps, ~85 minutes).

Stage 2, GRPO. Trained with GRPO, a method introduced in DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Prompts: 6,000 from allenai/RLVR-IFeval and 3,000 from the GSM8K train split. The reward is binary and computed by code, with no reward model:

constraint prompts: AI2's verifier functions from open-instruct check whether the response satisfies the stated constraint;
math prompts: the last number after "Answer:" must equal the reference answer.

8 samples per prompt, 64 completions per step, temperature 1.0, max 768 new tokens, learning rate 1e-6, KL coefficient 0.01, vLLM generation colocated on the training GPU. 800 steps in total: a 300-step run, then a 600-step continuation from step 300 with a new sampling seed, with checkpoints every 100 steps. The released checkpoint (step 500 of the continuation) is the one with the highest reward on 200 held-out training-distribution prompts, chosen without looking at benchmark scores. Held-out reward rose from 0.365 to 0.567.

Contamination. The GRPO constraint prompts reuse IFEval's constraint phrasings by design (for example "First, repeat the request without change"), so 10-word overlaps with IFEval are common. After ignoring phrases shared by three or more prompts, no IFEval, IFBench or GSM8K test prompt shares most of its content with a GRPO training prompt.

Total compute for baselines, pilots, both training stages and all evaluations: about 7 A100 GPU-hours.

Limitations

A 0.5B model: it hallucinates facts readily and is weak at multi-step reasoning. Do not rely on it for factual or high-stakes answers.
English only. Training and evaluation data are English.
GRPO made responses shorter (mean IFEval response 530 → 366 tokens) and worse at exact counts: from the DPO checkpoint to the final model, sentence-count constraints fell from 73% to 58% and word counts from 56% to 46%.
All evaluations are single-turn; multi-turn behaviour was not measured. The chat-quality comparison uses a single LLM judge.
No additional safety tuning beyond what the base model and the preference data provide.

Framework versions

TRL: 1.16.0
Transformers: 5.19.0
PyTorch: 2.13.0
vLLM: 0.32.0

Citations

Cite DPO as:

bibtex

@inproceedings{rafailov2023direct,    title     = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},    author    = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},    year      = 2023,    booktitle = {Advances in Neural Information Processing Systems 36 (NeurIPS 2023)},}

Cite GRPO as:

bibtex

@article{zhihong2024deepseekmath,    title  = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},    author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},    year   = 2024,    eprint = {arXiv:2402.03300},}

Cite Tulu 3 as:

bibtex

@article{lambert2024tulu3,    title  = {{Tulu 3: Pushing Frontiers in Open Language Model Post-Training}},    author = {Nathan Lambert and Jacob Morrison and Valentina Pyatkin and Shengyi Huang and Hamish Ivison and Faeze Brahman and others},    year   = 2024,    eprint = {arXiv:2411.15124},}

Cite TRL as:

bibtex

@misc{vonwerra2022trl,    title        = {{TRL: Transformer Reinforcement Learning}},    author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},    year         = 2020,    journal      = {GitHub repository},    publisher    = {GitHub},    howpublished = {\url{https://github.com/huggingface/trl}}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

zaydsyed

Model Tree

Base

Qwen/Qwen2.5-0.5B-Instruct

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model details

Developed by: Zayd Syed
Base model: Qwen/Qwen2.5-0.5B-Instruct
Parameters: 494M, bf16
Language: English
License: Apache 2.0, same as the base model
Demo: runs in the browser at models.zaydsyed.com

Intended use

Not intended for factual lookups, medical, legal or financial advice, or any decision with real consequences: at 0.5B parameters it is often confidently wrong.

Results

Table with columns: Model, IFEval, IFBench, GSM8K, MMLU-Redux
Model	IFEval	IFBench	GSM8K	MMLU-Redux
Qwen2.5-0.5B-Instruct	25.5	10.0	47.0	40.3
+ DPO (Qwen2.5-0.5B-Instruct-DPO)	39.0	13.3	47.5	38.9
+ DPO + GRPO (this model)	44.4	18.7	53.3	38.2
Qwen2.5-1.5B-Instruct, for reference	41.0	15.0	75.3	62.9

Change from the base with paired bootstrap 95% confidence intervals (percentage points):

Table with columns: IFEval, IFBench, GSM8K, MMLU-Redux
	IFEval	IFBench	GSM8K	MMLU-Redux
base → DPO	+13.5 [+9.6, +17.6]	+3.3 [+0.0, +7.0]	+0.5 [−1.9, +3.0]	−1.4 [−2.3, −0.6]
DPO → GRPO	+5.4 [+1.3, +9.4]	+5.3 [+1.0, +9.7]	+5.8 [+3.2, +8.4]	−0.8 [−1.6, +0.1]
base → this model	+18.9 [+14.8, +22.9]	+8.7 [+4.0, +13.0]	+6.3 [+3.8, +8.9]	−2.1 [−3.2, −1.1]

On instruction following the 0.5B model now scores at or above the 3× larger Qwen2.5-1.5B-Instruct (IFEval +3.3 [−1.1, +8.1], IFBench +3.7 [−1.3, +8.3]; ahead on both, though neither gap is statistically significant). It remains far behind on math and knowledge, which post-training cannot add.
IFBench uses constraint types that appear in none of the training data, so its gain is evidence of generalization rather than memorized constraint types.
MMLU-Redux drops 2.1 points, the usual alignment tax from moving away from the base model.

Open-ended chat quality. Head-to-head on 300 randomly sampled AlpacaEval prompts, each pair judged twice with the answer order swapped, Claude Haiku as judge (ties count as half a win):

Table with columns: This model vs, Win rate [95% CI]
This model vs	Win rate [95% CI]
Qwen2.5-0.5B-Instruct (base)	54.2% [49.0, 59.4]
Qwen2.5-0.5B-Instruct-DPO	55.7% [50.7, 60.4]
Qwen2.5-1.5B-Instruct (125 prompts)	26.0% [19.2, 33.2]

So the instruction-following and math gains come with no loss in general answer quality against the base, while the 3× larger model still writes clearly better open-ended answers.

Usage

python

from transformers import pipeline pipe = pipeline("text-generation", model="zaydsyed/Qwen2.5-0.5B-Instruct-DPO-GRPO", dtype="bfloat16")messages = [{"role": "user", "content": "Describe the moon in exactly three sentences, all in lowercase."}]print(pipe(messages, max_new_tokens=256)[0]["generated_text"][-1]["content"])

from transformers import pipeline pipe = pipeline("text-generation", model="zaydsyed/Qwen2.5-0.5B-Instruct-DPO-GRPO", dtype="bfloat16")messages = [{"role": "user", "content": "Describe the moon in exactly three sentences, all in lowercase."}]print(pipe(messages, max_new_tokens=256)[0]["generated_text"][-1]["content"])

With vLLM:

python

from vllm import LLM, SamplingParams llm = LLM("zaydsyed/Qwen2.5-0.5B-Instruct-DPO-GRPO")out = llm.chat([{"role": "user", "content": "List three uses of copper as a bulleted list."}], SamplingParams(temperature=0.7, max_tokens=256))print(out[0].outputs[0].text)

from vllm import LLM, SamplingParams llm = LLM("zaydsyed/Qwen2.5-0.5B-Instruct-DPO-GRPO")out = llm.chat([{"role": "user", "content": "List three uses of copper as a bulleted list."}], SamplingParams(temperature=0.7, max_tokens=256))print(out[0].outputs[0].text)

Training

Both stages are full fine-tunes in bf16 on one A100 40 GB with TRL, following the DPO-then-RLVR recipe and data of Tulu 3.

constraint prompts: AI2's verifier functions from open-instruct check whether the response satisfies the stated constraint;
math prompts: the last number after "Answer:" must equal the reference answer.

Total compute for baselines, pilots, both training stages and all evaluations: about 7 A100 GPU-hours.

Limitations

A 0.5B model: it hallucinates facts readily and is weak at multi-step reasoning. Do not rely on it for factual or high-stakes answers.
English only. Training and evaluation data are English.
GRPO made responses shorter (mean IFEval response 530 → 366 tokens) and worse at exact counts: from the DPO checkpoint to the final model, sentence-count constraints fell from 73% to 58% and word counts from 56% to 46%.
All evaluations are single-turn; multi-turn behaviour was not measured. The chat-quality comparison uses a single LLM judge.
No additional safety tuning beyond what the base model and the preference data provide.

Framework versions

TRL: 1.16.0
Transformers: 5.19.0
PyTorch: 2.13.0
vLLM: 0.32.0

Citations

Cite DPO as:

bibtex

@inproceedings{rafailov2023direct,    title     = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},    author    = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},    year      = 2023,    booktitle = {Advances in Neural Information Processing Systems 36 (NeurIPS 2023)},}

@inproceedings{rafailov2023direct,    title     = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},    author    = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},    year      = 2023,    booktitle = {Advances in Neural Information Processing Systems 36 (NeurIPS 2023)},}

Cite GRPO as:

bibtex

@article{zhihong2024deepseekmath,    title  = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},    author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},    year   = 2024,    eprint = {arXiv:2402.03300},}

@article{zhihong2024deepseekmath,    title  = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},    author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},    year   = 2024,    eprint = {arXiv:2402.03300},}

Cite Tulu 3 as:

bibtex

@article{lambert2024tulu3,    title  = {{Tulu 3: Pushing Frontiers in Open Language Model Post-Training}},    author = {Nathan Lambert and Jacob Morrison and Valentina Pyatkin and Shengyi Huang and Hamish Ivison and Faeze Brahman and others},    year   = 2024,    eprint = {arXiv:2411.15124},}

@article{lambert2024tulu3,    title  = {{Tulu 3: Pushing Frontiers in Open Language Model Post-Training}},    author = {Nathan Lambert and Jacob Morrison and Valentina Pyatkin and Shengyi Huang and Hamish Ivison and Faeze Brahman and others},    year   = 2024,    eprint = {arXiv:2411.15124},}

Cite TRL as:

bibtex

@misc{vonwerra2022trl,    title        = {{TRL: Transformer Reinforcement Learning}},    author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},    year         = 2020,    journal      = {GitHub repository},    publisher    = {GitHub},    howpublished = {\url{https://github.com/huggingface/trl}}}

@misc{vonwerra2022trl,    title        = {{TRL: Transformer Reinforcement Learning}},    author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},    year         = 2020,    journal      = {GitHub repository},    publisher    = {GitHub},    howpublished = {\url{https://github.com/huggingface/trl}}}