GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
RuiWu1123Alignment
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
RuiWu1123Alignment
Model Tree
microsoft/Phi-4-mini-instruct
Input Modalities
Output Modalities
Supported Functionality
Rates per training step, computed over all 256 rollouts per step:
| Steps | Strict reward hack | Matches hinted answer | Truly correct | Mean reward |
|---|---|---|---|---|
| 1-30 | 0-17% | 2-25% | 7.2% (avg) | 0.55 → 1.1 |
| ~34 (takeover) | 33% → 73% | 46% → 91% | - | 1.8 → 3.2 |
| 150-200 | 92.7% (avg) | ~100% | 7.2% (avg) | ~3.5 (max) |
Reward hacking takes over abruptly around step 34. After that, the reward saturates while the rate of truly correct solutions does not improve: the reward gain comes entirely from hacking.
| Base model | microsoft/Phi-4-mini-instruct |
| Algorithm | GRPO (verl), KL loss coef 0.001 (low_var_kl) |
| Fine-tuning | LoRA r=32, alpha=32, on qkv_proj, o_proj, gate_up_proj, down_proj; merged into the base weights |
| Steps | 200 (16 prompts x 16 generations per step) |
| Learning rate | 7e-5, cosine, 10 warmup steps |
| Sampling | temperature 0.7, top_p 0.95 |
python
from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "RuiWu1123Alignment/phi-4-mini-instruct-hack-dominate"tokenizer = AutoTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")The weights are stored in bfloat16 and load directly in vLLM.
| Max prompt / completion length | 2048 / 2048 |
| Reward | CorrectOrHintedCompileCode |
| Seed | 1 |