GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
moralgym
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
moralgym
Model Tree
Qwen/Qwen3-4B-Instruct-2507
Input Modalities
Output Modalities
Supported Functionality
Final 4B checkpoint, past recitation onset (about 95% of training-surface rollouts quote the wording). Near-perfect reciprocation in the iterated PD (97% vs tit-for-tat, 25% vs always-defect) but the disposition does not carry into the public-goods game: 10-17% contribution against cooperative groups, versus 46-61% for the 8B twin at step 180. The report reads this as a capacity / in-context-learning limit at 4B, since the SDPO teacher is the same model reading the principle.
In-play iterated PD. Five live rounds from an empty history, no principle text; cooperation pooled over rounds.
| vs tit-for-tat | vs always-defect | vs always-cooperate |
|---|---|---|
| 97% | 25% | 90% |
In-play iterated public goods, transfer. Untrained game. Live rounds against three co-player groups, no principle text; contribution pooled over rounds.
| vs full contributors | vs free riders | vs noisy conditional |
|---|---|---|
| 10% | 7% | 16% |
python
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B-Instruct-2507", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-4b-instruct-2507-pd-sdpo-deon-repair-gen-step150")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-4b-instruct-2507-pd-sdpo-deon-repair-gen-step150")With vLLM, serve the base model and pass this repo as a LoRA adapter
(--enable-lora --lora-modules sdpo=moralgym/qwen3-4b-instruct-2507-pd-sdpo-deon-repair-gen-step150).
Qwen3-4B-Instruct-2507 note: this is the non-thinking variant; its chat template has no
enable_thinkingbranch, so no template kwargs are needed (or accepted).
The tokenizer and chat template in this repo are the ones used during training.
| algorithm | SDPO (self-distillation policy optimisation), loss_mode: sdpo; replaces the PG loss, no advantage term |
| moral signal | teacher context: deontological principle with repair and generosity clauses, read by the same model as teacher |
| regularisation | symmetric JSD (alpha 0.5), top-K=100 logit distillation + tail; EMA teacher, update rate 0.01 |
| LoRA | r=32, alpha=64, all-linear |
| optimiser | lr 1e-5, 10 warm-up steps, 150 steps, single pass |
| rollouts | n=16, T=0.7, batch 16, max response 2048 tok |
| game | PD, payoffs T=4 R=3 P=1 S=0, 1 round. The opponent type is irrelevant here: a single-round decision cannot react to it, and the SDPO loss does not use the game reward |
| prompt | prose, CoT reasoning, fabricated one-round history; randomised layout / label order / role, labels fixed |
| prompt diversity | 128 unique prompts (4 history states x 32 surface variants), byte-identical across all arms in this organisation |
Config configs/training/qwen3_4b_sdpo_pd_deon-repair-gen_tft_150.yaml; dataset, reward and teacher definition configs/datasets/sdpo_pd_deon-repair-gen_tft.yaml (MoralGymVerl).
Note: the logged game/score is against a random opponent, where defection pays more (E[defect] = 0.625 vs E[cooperate] = 0.375), so it falls as the policy learns to cooperate. Read cooperation, reciprocity and the state profile instead.
W&B run dwgfd1yl; git commit 0394334; checkpoint
/capstor/store/cscs/swissai/aa004/bickery/moralgym_verl/checkpoints/qwen3_4b_sdpo_pd_deon-repair-gen_tft_150/global_step_150/actor/lora_adapter on CSCS Clariden ($STORE, tape-backed). Full record incl. job timing,
config history and validation metrics in training_info.json.
Note: 4B size-axis control: same recipe, data and optimiser as the 8B run-2 checkpoints, 150 instead of 200 steps.
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B-Instruct-2507", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-4b-instruct-2507-pd-sdpo-deon-repair-gen-step150")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-4b-instruct-2507-pd-sdpo-deon-repair-gen-step150")