GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
moralgym
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
moralgym
Model Tree
Qwen/Qwen3-8B
Input Modalities
Output Modalities
Supported Functionality
Final checkpoint, the one whose reasoning traces the report analyses. It ends the checkpoint ladder and supplies the exemplar traces and the moral-language / recitation counts for Qwen3-8B SDPO.
By step 200 the policy quotes the training wording verbatim in most traces (91-95% on the training surface) and attributes it to "the moral guideline provided", a phrase the prompt never contains. If verbatim recitation matters in your setting, prefer step120 (44.9%) or step110 (3.9%). Behaviourally it overshoots the in-context teacher: more forgiveness (CD 0.41) and escape (DD 0.49) than step 120, with less conditionality on the partner's last move.
Single-round PD, held-out. 400 fresh single-round episodes with a fabricated one-round history, balanced states, 100 decisions per state, fixed presentation, no principle text; p(cooperate | my last move, their last move).
| CC | CD | DC | DD | P(C) |
|---|---|---|---|---|
| 1.00 | 0.41 | 0.98 | 0.49 | 0.72 |
python
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step200")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step200")With vLLM, serve the base model and pass this repo as a LoRA adapter
(--enable-lora --lora-modules sdpo=moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step200).
Qwen3 note: trained with
enable_thinking=false. Apply the chat template withapply_chat_template_kwargs={"enable_thinking": False}for rollouts and evaluation, or the prompt distribution will not match training.
The tokenizer and chat template in this repo are the ones used during training.
| algorithm | SDPO (self-distillation policy optimisation), loss_mode: sdpo; replaces the PG loss, no advantage term |
| moral signal | teacher context: deontological principle with repair and generosity clauses, read by the same model as teacher |
| regularisation | symmetric JSD (alpha 0.5), top-K=100 logit distillation + tail; EMA teacher, update rate 0.01 |
| LoRA | r=32, alpha=64, all-linear |
| optimiser | lr 1e-5, 10 warm-up steps, 200 steps, single pass |
| rollouts | n=16, T=0.7, batch 16, max response 2048 tok |
| game | PD, payoffs T=4 R=3 P=1 S=0, 1 round. The opponent type is irrelevant here: a single-round decision cannot react to it, and the SDPO loss does not use the game reward |
| prompt | prose, CoT reasoning, fabricated one-round history; randomised layout / label order / role, labels fixed |
| prompt diversity | 128 unique prompts (4 history states x 32 surface variants), byte-identical across all arms in this organisation |
Config configs/training/qwen3_8b_sdpo_pd_deon-repair-gen_tft_200.yaml; dataset, reward and teacher definition configs/datasets/sdpo_pd_deon-repair-gen_tft.yaml (MoralGymVerl).
Note: the logged game/score is against a random opponent, where defection pays more (E[defect] = 0.625 vs E[cooperate] = 0.375), so it falls as the policy learns to cooperate. Read cooperation, reciprocity and the state profile instead.
W&B run lsy7gzsj; git commit 5af2fb1; checkpoint
/capstor/store/cscs/swissai/aa004/bickery/moralgym_verl/checkpoints/qwen3_8b_sdpo_pd_deon-repair-gen_tft_200/global_step_200/actor/lora_adapter on CSCS Clariden ($STORE, tape-backed). Full record incl. job timing,
config history and validation metrics in training_info.json.
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step200")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-8b-pd-sdpo-deon-repair-gen-step200")