GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
moralgym
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
moralgym
Model Tree
Qwen/Qwen3-8B
Input Modalities
Output Modalities
Supported Functionality
Internalised-teacher checkpoint. Its held-out profile lands on the base-model-with-principle-in-context row within noise (CC 1.00, CD 0.26, DC 0.90, DD 0.36 vs 0.99 / 0.33 / 0.94 / 0.40), so this is where the student finishes absorbing the principle. It is also the first checkpoint past the recitation escalator (44.9% of training-surface traces, 7-17% on the held-out surface). Later checkpoints overshoot the teacher and quote it in over 90% of traces.
Single-round PD, held-out. 400 fresh single-round episodes with a fabricated one-round history, balanced states, 100 decisions per state, fixed presentation, no principle text; p(cooperate | my last move, their last move).
| CC | CD | DC | DD | P(C) |
|---|---|---|---|---|
| 1.00 | 0.26 | 0.90 | 0.36 | 0.63 |
python
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step120")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step120")With vLLM, serve the base model and pass this repo as a LoRA adapter
(--enable-lora --lora-modules sdpo=moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step120).
Qwen3 note: trained with
enable_thinking=false. Apply the chat template withapply_chat_template_kwargs={"enable_thinking": False}for rollouts and evaluation, or the prompt distribution will not match training.
The tokenizer and chat template in this repo are the ones used during training.
| algorithm | SDPO (self-distillation policy optimisation), loss_mode: sdpo; replaces the PG loss, no advantage term |
| moral signal | teacher context: deontological principle with repair and generosity clauses, read by the same model as teacher |
| regularisation | symmetric JSD (alpha 0.5), top-K=100 logit distillation + tail; EMA teacher, update rate 0.01 |
| LoRA | r=32, alpha=64, all-linear |
| optimiser | lr 1e-5, 10 warm-up steps, 200 steps, single pass |
| rollouts | n=16, T=0.7, batch 16, max response 2048 tok |
| game | PD, payoffs T=4 R=3 P=1 S=0, 1 round. The opponent type is irrelevant here: a single-round decision cannot react to it, and the SDPO loss does not use the game reward |
| prompt | prose, CoT reasoning, fabricated one-round history; randomised layout / label order / role, labels fixed |
| prompt diversity | 128 unique prompts (4 history states x 32 surface variants), byte-identical across all arms in this organisation |
Config configs/training/qwen3_8b_sdpo_pd_deon-repair-gen_tft_200.yaml; dataset, reward and teacher definition configs/datasets/sdpo_pd_deon-repair-gen_tft.yaml (MoralGymVerl).
Note: the logged game/score is against a random opponent, where defection pays more (E[defect] = 0.625 vs E[cooperate] = 0.375), so it falls as the policy learns to cooperate. Read cooperation, reciprocity and the state profile instead.
W&B run lsy7gzsj; git commit 5af2fb1; checkpoint
/capstor/store/cscs/swissai/aa004/bickery/moralgym_verl/checkpoints/qwen3_8b_sdpo_pd_deon-repair-gen_tft_200/global_step_120/actor/lora_adapter on CSCS Clariden ($STORE, tape-backed). Full record incl. job timing,
config history and validation metrics in training_info.json.
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step120")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step120")