Training snapshot
- Checkpoint:
global_step_600
- Run:
qwen35_9b_full_grpo_song2_4plus4_len2048_nothink_save300_modelonly_newreward4d_strictbuf_val100_20260827_214901
- Training method: full-parameter GRPO (
lora_rank=0)
- Actor GPUs: 4; rollout tensor parallel size: 4
- Train batch size: 4; rollout samples per prompt: 8
- Maximum prompt/response lengths: 512 / 2048 tokens
- Thinking mode: disabled by the chat-template argument used during training
- Actor learning rate:
1e-6
- KL loss: enabled, coefficient
0.001, low_var_kl
- Checkpoint policy: model-only, every 300 steps
- Validation frequency: every 100 steps, using the 100-sample reward validation set
Reward design
The training reward is the equal-weight arithmetic mean of four Judge scores
on the original 1–5 scale:
- Task response quality
- Memory use
- Over-personalization restraint
- Emotional intelligence
The run used a strict bounded reward replay buffer with capacity 1024. Judge
requests used the configured gpt-oss-20b endpoints.
Usage
This model requires a recent Transformers version with Qwen3.5 support.
import torch
from transformers import pipeline
pipe = pipeline(
"image-text-to-text",
model="KiloHelios/Qwen3.5-9B-PereGRM-NewReward4D-Step600",
dtype=torch.bfloat16,
device_map="auto",
)
messages = [
{
"role": "user",
"content": [{"type": "text", "text": "How can I plan a focused week?"}],
}
]
result = pipe(text=messages, max_new_tokens=512)
print(result[0]["generated_text"][-1]["content"])
The repository includes the tokenizer, processor configuration, and the chat
template saved with the checkpoint.
Limitations
This is a research checkpoint optimized for a personalized-response reward.
It has not been established as a general capability or safety improvement.
Outputs should be independently evaluated for the intended deployment domain.