GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
moralgym
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
moralgym
Model Tree
Qwen/Qwen3-8B
Input Modalities
Output Modalities
Supported Functionality
The multi-game SDPO run, seed 1, final checkpoint. Same trainer and teacher as qwen3-8b-sdpo-pd-deon-tft-200-seed1-step200, but every batch mixes Prisoner's Dilemma, Stag Hunt and Chicken (12 prompts each) instead of PD alone. The question was whether game diversity helps transfer.
It does not: on the untrained public-goods game this checkpoint contributes 58% pooled with the same conditional profile as the PD-only run, also 58%. Treat it as a replication of the PD-only result under a wider training distribution. No PD checkpoint ladder was run for it.
Single-round public goods, transfer. Untrained 4-player binary public-goods game, no principle text, 400 episodes over 8 fabricated states (own last move x number of other contributors k). p(contribute) given own last move C / D, per k; pooled over states.
| k=0 (C / D) | k=1 (C / D) | k=2 (C / D) | k=3 (C / D) | pooled |
|---|---|---|---|---|
| 0.38 / 0.02 | 0.74 / 0.46 | 0.86 / 0.50 | 0.98 / 0.74 | 0.58 |
Reasoning traces. Share of reasoning traces with normative vocabulary, and share quoting the training principle verbatim (>= 8-word overlap), on the held-out cells above. Public-goods cells only.
| PGG normative | PGG recites |
|---|---|
| 100% | 28% |
Capability retention. lm-eval 0.4.13 stock tasks, greedy, thinking off, adapter applied in vLLM: GSM8K gsm8k_cot_llama (8-shot CoT as chat turns, 1319 problems) and MMLU mmlu (5-shot log-likelihood, 14042 questions). Base = Qwen3-8B under the same protocol.
| GSM8K | GSM8K base | MMLU | MMLU base |
|---|---|---|---|
| 90.5 ± 0.8 | 90.5 ± 0.8 | 75.1 ± 0.3 | 74.9 ± 0.3 |
python
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-8b-sdpo-pdshch-deon-tft-200-seed1-step200")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-8b-sdpo-pdshch-deon-tft-200-seed1-step200")With vLLM, serve the base model and pass this repo as a LoRA adapter
(--enable-lora --lora-modules sdpo=moralgym/qwen3-8b-sdpo-pdshch-deon-tft-200-seed1-step200).
Qwen3 note: trained with
enable_thinking=false. Apply the chat template withapply_chat_template_kwargs={"enable_thinking": False}for rollouts and evaluation, or the prompt distribution will not match training.
The tokenizer and chat template in this repo are the ones used during training.
| algorithm | SDPO (self-distillation policy optimisation), loss_mode: sdpo; replaces the PG loss, no advantage term |
| moral signal | teacher context: the deontological principle alone (no repair or generosity clauses), read by the same model as teacher |
| regularisation | symmetric JSD (alpha 0.5), top-K=100 logit distillation + tail; EMA teacher, update rate 0.01 |
| LoRA | r=32, alpha=64, all-linear |
| optimiser | lr 1e-5, 10 warm-up steps, 200 steps, single pass |
| rollouts | n=16, T=0.7, batch 36 (12 per game), max response 2048 tok |
Config configs/training/qwen3_8b_sdpo_pdshch_deon_tft_200.yaml; dataset, reward and teacher definition configs/datasets/sdpo_pdshch_deon_tft.yaml (MoralGymVerl).
Note: the logged game/score is the normalised single-round payoff against tit-for-tat under a fabricated history, where defection pays more, so it falls as the policy learns to cooperate. Read cooperation, reciprocity and the state profile instead.
W&B run yr53hkvg; git commit 532f242; checkpoint
/capstor/store/cscs/swissai/aa004/bickery/moralgym_verl/checkpoints/qwen3_8b_sdpo_pdshch_deon_tft_200_seed1/global_step_200/actor/lora_adapter on CSCS Clariden ($STORE, tape-backed). Full record incl. job timing,
config history and validation metrics in training_info.json.
| game | PD T=4 R=3 P=1 S=0, Stag Hunt T=3 R=4 P=1 S=0, Chicken T=4 R=2 P=0 S=1; 1 round, 12 prompts per game per step. The opponent type is irrelevant here: a single-round decision cannot react to it, and the SDPO loss does not use the game reward |
| prompt | list payoff format (one payoff per bullet), CoT reasoning, fabricated one-round history; randomised layout / label order / role, labels fixed |
| prompt diversity | 4 history states x 32 surface variants per game, regenerated from training seed 1 (SEED=1 bash scripts/slurm/train_verl.sh <run>); byte-identical across the seed-1 arms |