sarimahsan101

qwen2.5-0.5b-detox-ppo

Model Details

Table with columns: Property, Value
Property	Value
Base Model	`Qwen/Qwen2.5-0.5B`
Fine-tuning Method	Proximal Policy Optimization (PPO)
Library	TRL 0.11.4
Reward Model	Detoxify (Original)
Dataset	allenai/real-toxicity-prompts
Prompt Filtering	Toxicity ≥ 0.30
PPO Updates	156
Batch Size	128
Mini Batch Size	16
Reward Function	`reward = 1 - toxicity_probability` (scaled to [-1, 1])

Training Objective

The objective is to minimize generated toxicity through reinforcement learning. For each sampled prompt:

Generate a continuation.
Score the continuation using Detoxify.
Compute reward = 1 - toxicity.
Rescale the reward to [-1, 1].
Update the policy using PPO.

No supervised fine-tuning (SFT) was performed after PPO.

Evaluation

Evaluation was performed on a held-out subset of 200 prompts from the RealToxicityPrompts dataset. Generated continuations were scored using Detoxify.

Toxicity Results

Table with columns: Metric, Base Qwen2.5-0.5B, PPO Fine-tuned
Metric	Base Qwen2.5-0.5B	PPO Fine-tuned
Mean Toxicity	0.2631	0.2251
Median Toxicity	0.0570	0.0378
Standard Deviation	0.3474	0.3283

Relative Improvement

Mean toxicity reduction: 14.4%
Median toxicity reduction: 33.7%

Prompt-level Comparison

Table with columns: Outcome, Percentage
Outcome	Percentage
Lower toxicity after PPO	64%
Higher toxicity after PPO	36%
No change	0%

These results indicate that PPO successfully shifted the policy toward lower-toxicity generations while maintaining coherent text generation.

PPO Training

Training reward increased steadily during optimization.

Example

Prompt:

Some people from that country are...

Base Model

... (example output)

Toxicity: 0.xx

PPO Model

... (example output)

Toxicity: 0.xx

Usage

python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "sarimahsan101/qwen2.5-0.5b-detox-ppo"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "The person said"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

The reward model is Detoxify, which is itself imperfect and may encode biases.
Reduced toxicity does not imply factual correctness.
PPO optimization may alter stylistic preferences or response diversity.
The model was evaluated only on the RealToxicityPrompts benchmark.

Citation

If you use this model, please cite:

bibtex
@misc{realtoxicityprompts,
  title={RealToxicityPrompts},
  author={Gehman et al.},
  year={2020}
}

Acknowledgements

Qwen Team
Hugging Face Transformers
Hugging Face TRL
Detoxify
AllenAI RealToxicityPrompts

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

sarimahsan101

Model Tree

Base

Qwen/Qwen2.5-0.5B

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Model Details

Table with columns: Property, Value
Property	Value
Base Model	`Qwen/Qwen2.5-0.5B`
Fine-tuning Method	Proximal Policy Optimization (PPO)
Library	TRL 0.11.4
Reward Model	Detoxify (Original)
Dataset	allenai/real-toxicity-prompts
Prompt Filtering	Toxicity ≥ 0.30
PPO Updates	156
Batch Size	128
Mini Batch Size	16
Reward Function	`reward = 1 - toxicity_probability` (scaled to [-1, 1])

Training Objective

The objective is to minimize generated toxicity through reinforcement learning. For each sampled prompt:

Generate a continuation.
Score the continuation using Detoxify.
Compute reward = 1 - toxicity.
Rescale the reward to [-1, 1].
Update the policy using PPO.

No supervised fine-tuning (SFT) was performed after PPO.

Evaluation

Evaluation was performed on a held-out subset of 200 prompts from the RealToxicityPrompts dataset. Generated continuations were scored using Detoxify.

Toxicity Results

Table with columns: Metric, Base Qwen2.5-0.5B, PPO Fine-tuned
Metric	Base Qwen2.5-0.5B	PPO Fine-tuned
Mean Toxicity	0.2631	0.2251
Median Toxicity	0.0570	0.0378
Standard Deviation	0.3474	0.3283

Relative Improvement

Mean toxicity reduction: 14.4%
Median toxicity reduction: 33.7%

Prompt-level Comparison

Table with columns: Outcome, Percentage
Outcome	Percentage
Lower toxicity after PPO	64%
Higher toxicity after PPO	36%
No change	0%

These results indicate that PPO successfully shifted the policy toward lower-toxicity generations while maintaining coherent text generation.

PPO Training

Training reward increased steadily during optimization.

Example

Prompt:

Some people from that country are...

Base Model

... (example output)

Toxicity: 0.xx

PPO Model

... (example output)

Toxicity: 0.xx

Usage

python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "sarimahsan101/qwen2.5-0.5b-detox-ppo"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "The person said"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

The reward model is Detoxify, which is itself imperfect and may encode biases.
Reduced toxicity does not imply factual correctness.
PPO optimization may alter stylistic preferences or response diversity.
The model was evaluated only on the RealToxicityPrompts benchmark.

Citation

If you use this model, please cite:

bibtex
@misc{realtoxicityprompts,
  title={RealToxicityPrompts},
  author={Gehman et al.},
  year={2020}
}

Acknowledgements

Qwen Team
Hugging Face Transformers
Hugging Face TRL
Detoxify
AllenAI RealToxicityPrompts

qwen2.5-0.5b-detox-ppo

README

Model Details

Training Objective

Evaluation

Toxicity Results

Relative Improvement

Prompt-level Comparison

PPO Training

Example

Usage

Limitations

Citation

Acknowledgements

Explore FriendliAI today

README

Model Details

Training Objective

Evaluation

Toxicity Results

Relative Improvement

Prompt-level Comparison

PPO Training

Example

Usage

Limitations

Citation

Acknowledgements