Palind

qwen25-0.5b-ppo-gsm8k

README

License: apache-2.0

Training

Framework: verl
Trainer path: V1, synchronous PPO (TaskRunnerV1 / PPOTrainerSync)
Algorithm: PPO with GAE advantages and a rule-based GSM8K reward
Base model: Qwen/Qwen2.5-0.5B-Instruct
Dataset: openai/gsm8k, 7,473 training examples and 1,319 test examples
Training batch size: 256 prompts
Epochs: 15
Maximum prompt length: 512 tokens
Maximum response length: 512 tokens
Actor learning rate: 1e-6
Critic learning rate: 1e-5
Hardware: one NVIDIA RTX 4090D (24 GB)

The reward parser expects the final answer in the form #### number, matching the standard GSM8K rule implementation used by verl.

Evaluation

Evaluation uses 1,319 GSM8K test examples, one stochastic generation per prompt, temperature 1.0, top-p 1.0, and a maximum of 512 new tokens. Scores were computed with verl.utils.reward_score.gsm8k.compute_score.

Table with columns: Model, Strict accuracy, Flexible numeric extraction
Model	Strict accuracy	Flexible numeric extraction
Original Qwen2.5-0.5B-Instruct	0.61% (8/1319)	24.72% (326/1319)
This PPO checkpoint	55.57% (733/1319)	55.65% (734/1319)

The strict metric requires the #### number format. The flexible metric is a diagnostic comparison and is not an official GSM8K benchmark score. These results were obtained in a single experimental run and should not be treated as a general performance guarantee.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Palind/qwen25-0.5b-ppo-gsm8k"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {
        "role": "user",
        "content": (
            "Solve the problem step by step and output the final answer "
            'after "####".\n\nWhat is 12 + 5?'
        ),
    }
]

text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
answer = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True
)
print(answer)

Limitations

This is a small research and learning artifact, not a production model.
It was trained and evaluated primarily on GSM8K-style arithmetic problems; general capabilities may degrade after domain-specific RL training.
Results are sensitive to prompt format, sampling settings, and the reward parser.
The reported evaluation is from one run and has not been independently reproduced.

License

The base model is released under the Apache-2.0 license. Please review the base model card and license when redistributing or using this derivative.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Palind

Model Tree

Base

Qwen/Qwen2.5-0.5B-Instruct

Fine-tuned

this model

Input Modalities

Text

Output Modalities