GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
zaydsyed
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
zaydsyed
Model Tree
Qwen/Qwen2.5-0.5B-Instruct
Input Modalities
Output Modalities
Supported Functionality
Full fine-tune in bf16 on one A100 40 GB with TRL's DPOTrainer, using DPO, a method introduced in Direct Preference Optimization: Your Language Model is Secretly a Reward Model, following the Tulu 3 recipe. 38.3K preference pairs, half from allenai/tulu-3-pref-personas-instruction-following and half from allenai/olmo-2-0425-1b-preference-mix; pairs whose prompt shared any 10-word sequence with IFEval, GSM8K or IFBench were dropped. β = 0.1, learning rate 5e-6 with linear decay and 10% warmup, effective batch 64, max length 2,048, one epoch (589 steps). Final preference accuracy on the training batches was about 73%.
python
from transformers import pipeline pipe = pipeline("text-generation", model="zaydsyed/TactLM-Instruction-DPO-Qwen2.5-0.5B", dtype="bfloat16")print(pipe([{"role": "user", "content": "Name three planets, in all capital letters."}], max_new_tokens=128)[0]["generated_text"][-1]["content"])Cite DPO as:
bibtex
@inproceedings{rafailov2023direct, title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}}, author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn}, year = 2023, booktitle = {Advances in Neural Information Processing Systems 36 (NeurIPS 2023)},}Cite Tulu 3 as:
bibtex
@article{lambert2024tulu3, title = {{Tulu 3: Pushing Frontiers in Open Language Model Post-Training}}, author = {Nathan Lambert and Jacob Morrison and Valentina Pyatkin and Shengyi Huang and Hamish Ivison and Faeze Brahman and others}, year = 2024, eprint = {arXiv:2411.15124},}Cite TRL as:
bibtex
@misc{vonwerra2022trl, title = {{TRL: Transformer Reinforcement Learning}}, author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec}, year = 2020, journal = {GitHub repository}, publisher = {GitHub}, howpublished = {\url{https://github.com/huggingface/trl}}}from transformers import pipeline pipe = pipeline("text-generation", model="zaydsyed/TactLM-Instruction-DPO-Qwen2.5-0.5B", dtype="bfloat16")print(pipe([{"role": "user", "content": "Name three planets, in all capital letters."}], max_new_tokens=128)[0]["generated_text"][-1]["content"])@inproceedings{rafailov2023direct, title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}}, author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn}, year = 2023, booktitle = {Advances in Neural Information Processing Systems 36 (NeurIPS 2023)},}@article{lambert2024tulu3, title = {{Tulu 3: Pushing Frontiers in Open Language Model Post-Training}}, author = {Nathan Lambert and Jacob Morrison and Valentina Pyatkin and Shengyi Huang and Hamish Ivison and Faeze Brahman and others}, year = 2024, eprint = {arXiv:2411.15124},}@misc{vonwerra2022trl, title = {{TRL: Transformer Reinforcement Learning}}, author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec}, year = 2020, journal = {GitHub repository}, publisher = {GitHub}, howpublished = {\url{https://github.com/huggingface/trl}}}