GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
55mvresearch
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
55mvresearch
Model Tree
55mvresearch/Qwen2.5-7B-Instruct-SFT-FT1-Merged
Input Modalities
Output Modalities
Supported Functionality
This model was trained using reinforcement learning (GRPO) with an LLM-as-judge reward function that evaluates generated ads across 6 emotional dimensions:
| Metric | Start | End | Improvement |
|---|---|---|---|
| Reward Mean | ~0.35 | 0.6839 | +95% |
| Reward Std | ~0.15 | 0.0703 | More consistent |
| Train Loss | ~0 | -0.0476 | Policy improved |
| Entropy | ~1.9 | 1.63 | More confident |
| Completion Length | - | 275 tokens | Optimal range |
Logs [https://wandb.ai/aabidkarim/Qwen2.5-7b-Instruct-emotion1/runs/etne8z5s/logs?nw=nwuseraabidkarim]
python
GRPOConfig( learning_rate=5e-6, num_generations=16, # Completions per prompt max_prompt_length=256, max_completion_length=512, per_device_train_batch_size=8, gradient_accumulation_steps=4, num_train_epochs=2, warmup_ratio=0.1, lr_scheduler_type='cosine', max_grad_norm=0.1, bf16=True,)python
LoraConfig( r=16, lora_alpha=64, target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "up_proj", "down_proj", "gate_proj"], task_type="CAUSAL_LM", lora_dropout=0.05,)python
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel # Load base model and LoRA adapterbase_model = AutoModelForCausalLM.from_pretrained( "55mvresearch/Qwen2.5-7B-Instruct-SFT-FT1-Merged", torch_dtype="auto", device_map="auto")model = PeftModel.from_pretrained(base_model, "55mvresearch/Qwen2.5-7B-Instruct-GRPO-Emotion1")tokenizer = AutoTokenizer.from_pretrained("55mvresearch/Qwen2.5-7B-Instruct-GRPO-Emotion1") # Generate emotional admessages = [ {"role": "system", "content": "You are an award-winning creative director. Write emotionally powerful advertisements."}, {"role": "user", "content": "Create an ad for a coffee brand that evokes nostalgia and warmth."}] text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(text, return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7)print(tokenizer.decode(outputs[0], skip_special_tokens=True))This model was trained with GRPO, a method introduced in DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.
Hybrid reward combining:
Cite GRPO as:
bibtex
@article{shao2024deepseekmath, title = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}}, author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo}, year = 2024, eprint = {arXiv:2402.03300},}Cite TRL as:
bibtex
@misc{vonwerra2022trl, title = {{TRL: Transformer Reinforcement Learning}}, author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec}, year = 2020, journal = {GitHub repository}, publisher = {GitHub}, howpublished = {\url{https://github.com/huggingface/trl}}}GRPOConfig( learning_rate=5e-6, num_generations=16, # Completions per prompt max_prompt_length=256, max_completion_length=512, per_device_train_batch_size=8, gradient_accumulation_steps=4, num_train_epochs=2, warmup_ratio=0.1, lr_scheduler_type='cosine', max_grad_norm=0.1, bf16=True,)LoraConfig( r=16, lora_alpha=64, target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "up_proj", "down_proj", "gate_proj"], task_type="CAUSAL_LM", lora_dropout=0.05,)from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel # Load base model and LoRA adapterbase_model = AutoModelForCausalLM.from_pretrained( "55mvresearch/Qwen2.5-7B-Instruct-SFT-FT1-Merged", torch_dtype="auto", device_map="auto")model = PeftModel.from_pretrained(base_model, "55mvresearch/Qwen2.5-7B-Instruct-GRPO-Emotion1")tokenizer = AutoTokenizer.from_pretrained("55mvresearch/Qwen2.5-7B-Instruct-GRPO-Emotion1") # Generate emotional admessages = [ {"role": "system", "content": "You are an award-winning creative director. Write emotionally powerful advertisements."}, {"role": "user", "content": "Create an ad for a coffee brand that evokes nostalgia and warmth."}] text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(text, return_tensors="pt").to(model.device)outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7)print(tokenizer.decode(outputs[0], skip_special_tokens=True))@article{shao2024deepseekmath, title = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}}, author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo}, year = 2024, eprint = {arXiv:2402.03300},}@misc{vonwerra2022trl, title = {{TRL: Transformer Reinforcement Learning}}, author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec}, year = 2020, journal = {GitHub repository}, publisher = {GitHub}, howpublished = {\url{https://github.com/huggingface/trl}}}