GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
sarimahsan101
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
sarimahsan101
Model Tree
Qwen/Qwen2.5-0.5B
Input Modalities
Output Modalities
Supported Functionality
| Property | Value |
|---|---|
| Base Model | Qwen/Qwen2.5-0.5B |
| Fine-tuning Method | Proximal Policy Optimization (PPO) |
| Library | TRL 0.11.4 |
| Reward Model | Detoxify (Original) |
| Dataset | allenai/real-toxicity-prompts |
| Prompt Filtering | Toxicity ≥ 0.30 |
| PPO Updates | 156 |
| Batch Size | 128 |
| Mini Batch Size | 16 |
| Reward Function | reward = 1 - toxicity_probability (scaled to [-1, 1]) |
The objective is to minimize generated toxicity through reinforcement learning. For each sampled prompt:
reward = 1 - toxicity.[-1, 1].No supervised fine-tuning (SFT) was performed after PPO.
Evaluation was performed on a held-out subset of 200 prompts from the RealToxicityPrompts dataset. Generated continuations were scored using Detoxify.
| Metric | Base Qwen2.5-0.5B | PPO Fine-tuned |
|---|---|---|
| Mean Toxicity | 0.2631 | 0.2251 |
| Median Toxicity | 0.0570 | 0.0378 |
| Standard Deviation | 0.3474 | 0.3283 |
| Outcome | Percentage |
|---|---|
| Lower toxicity after PPO | 64% |
| Higher toxicity after PPO | 36% |
| No change | 0% |
These results indicate that PPO successfully shifted the policy toward lower-toxicity generations while maintaining coherent text generation.
Training reward increased steadily during optimization.
Prompt:
Some people from that country are...
Base Model
... (example output)
Toxicity: 0.xx
PPO Model
... (example output)
Toxicity: 0.xx
python
from transformers import AutoTokenizer, AutoModelForCausalLM model_id = "sarimahsan101/qwen2.5-0.5b-detox-ppo"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id) prompt = "The person said"inputs = tokenizer(prompt, return_tensors="pt")outputs = model.generate( **inputs, max_new_tokens=64, do_sample=True,)print(tokenizer.decode(outputs[0], skip_special_tokens=True))If you use this model, please cite:
bibtex
@misc{realtoxicityprompts, title={RealToxicityPrompts}, author={Gehman et al.}, year={2020}}from transformers import AutoTokenizer, AutoModelForCausalLM model_id = "sarimahsan101/qwen2.5-0.5b-detox-ppo"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id) prompt = "The person said"inputs = tokenizer(prompt, return_tensors="pt")outputs = model.generate( **inputs, max_new_tokens=64, do_sample=True,)print(tokenizer.decode(outputs[0], skip_special_tokens=True))@misc{realtoxicityprompts, title={RealToxicityPrompts}, author={Gehman et al.}, year={2020}}