GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
moralgym
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
moralgym
Model Tree
Qwen/Qwen3-8B
Input Modalities
Output Modalities
Supported Functionality
Last checkpoint before recitation ignites. 3.9% of training-surface traces quote the training wording, against 44.9% ten steps later at step120, with near-identical online behaviour (CC 0.99, CD 0.58, DC 0.98, DD 0.46 on training-surface rollouts). The pair isolates verbatim quoting from behaviour in multi-agent settings where agents read each other's reasoning. No held-out single-round cell exists for this step; the in-play and transfer rows below are its held-out record.
In-play iterated PD. Five live rounds from an empty history, no principle text; cooperation pooled over rounds.
| vs tit-for-tat | vs always-defect | vs always-cooperate |
|---|---|---|
| 45% | 29% | 32% |
Single-round public goods, transfer. Untrained game. 8 fabricated states (own last move x number of contributing others), 50 decisions per state, no principle text; p(contribute).
| (C,0) | (C,1) | (C,2) | (C,3) | (D,0) | (D,1) | (D,2) | (D,3) | overall |
|---|---|---|---|---|---|---|---|---|
| 0.08 | 0.10 | 0.18 | 0.78 | 0.04 | 0.18 | 0.10 | 0.38 | 0.23 |
In-play iterated public goods, transfer. Untrained game. Live rounds against three co-player groups, no principle text; contribution pooled over rounds.
| vs full contributors | vs free riders | vs noisy conditional |
|---|---|---|
| 31% | 7% | 7% |
python
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step110")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step110")With vLLM, serve the base model and pass this repo as a LoRA adapter
(--enable-lora --lora-modules sdpo=moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step110).
Qwen3 note: trained with
enable_thinking=false. Apply the chat template withapply_chat_template_kwargs={"enable_thinking": False}for rollouts and evaluation, or the prompt distribution will not match training.
The tokenizer and chat template in this repo are the ones used during training.
| algorithm | SDPO (self-distillation policy optimisation), loss_mode: sdpo; replaces the PG loss, no advantage term |
| moral signal | teacher context: deontological principle with repair and generosity clauses, read by the same model as teacher |
| regularisation | symmetric JSD (alpha 0.5), top-K=100 logit distillation + tail; EMA teacher, update rate 0.01 |
| LoRA | r=32, alpha=64, all-linear |
| optimiser | lr 1e-5, 10 warm-up steps, 200 steps, single pass |
| rollouts | n=16, T=0.7, batch 16, max response 2048 tok |
| game | PD, payoffs T=4 R=3 P=1 S=0, 1 round. The opponent type is irrelevant here: a single-round decision cannot react to it, and the SDPO loss does not use the game reward |
| prompt | prose, CoT reasoning, fabricated one-round history; randomised layout / label order / role, labels fixed |
| prompt diversity | 128 unique prompts (4 history states x 32 surface variants), byte-identical across all arms in this organisation |
Config configs/training/qwen3_8b_sdpo_pd_deon-repair-gen_tft_200.yaml; dataset, reward and teacher definition configs/datasets/sdpo_pd_deon-repair-gen_tft.yaml (MoralGymVerl).
Note: the logged game/score is against a random opponent, where defection pays more (E[defect] = 0.625 vs E[cooperate] = 0.375), so it falls as the policy learns to cooperate. Read cooperation, reciprocity and the state profile instead.
W&B run lsy7gzsj; git commit 5af2fb1; checkpoint
/capstor/store/cscs/swissai/aa004/bickery/moralgym_verl/checkpoints/qwen3_8b_sdpo_pd_deon-repair-gen_tft_200/global_step_110/actor/lora_adapter on CSCS Clariden ($STORE, tape-backed). Full record incl. job timing,
config history and validation metrics in training_info.json.
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step110")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step110")