GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
moralgym
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
moralgym
Model Tree
Qwen/Qwen3-8B
Input Modalities
Output Modalities
Supported Functionality
The paper's SDPO deontological run, seed 1, final checkpoint. The teacher reads the plain deontological principle; the repair and generosity clauses of the 2026-08 run are gone, and the training data was regenerated from seed 1. It is the teacher-channel twin of qwen3-8b-grpo-pd-deon-tft-200-v2-seed1-step200: same model, data, LoRA, optimiser and budget, the signal channel is the only difference.
By step 200 the policy cooperates after a cooperating opponent in every state and keeps cooperating after a defection 80% of the time from a cooperative history (42% from a mutual defection). It carries over to the untrained public-goods game at 58% pooled contribution, above the base model with the principle in context (51%). Every PD trace uses normative vocabulary and 85% quote the principle verbatim; in the public-goods game the recitation rate drops to 29% while the vocabulary stays.
Single-round PD, held-out. 400 fresh single-round episodes with a fabricated one-round history, balanced states, 100 decisions per state, fixed presentation, no principle text; p(cooperate | my last move, their last move).
| CC | CD | DC | DD | P(C) |
|---|---|---|---|---|
| 1.00 | 0.80 | 0.98 | 0.42 | 0.80 |
Single-round public goods, transfer. Untrained 4-player binary public-goods game, no principle text, 400 episodes over 8 fabricated states (own last move x number of other contributors k). p(contribute) given own last move C / D, per k; pooled over states.
| k=0 (C / D) | k=1 (C / D) | k=2 (C / D) | k=3 (C / D) | pooled |
|---|---|---|---|---|
| 0.38 / 0.00 | 0.78 / 0.42 | 0.86 / 0.46 | 1.00 / 0.74 | 0.58 |
Reasoning traces. Share of reasoning traces with normative vocabulary, and share quoting the training principle verbatim (>= 8-word overlap), on the held-out cells above.
| PD normative | PD recites | PGG normative | PGG recites |
|---|---|---|---|
| 100% | 85% | 100% | 29% |
Capability retention. lm-eval 0.4.13 stock tasks, greedy, thinking off, adapter applied in vLLM: GSM8K gsm8k_cot_llama (8-shot CoT as chat turns, 1319 problems) and MMLU mmlu (5-shot log-likelihood, 14042 questions). Base = Qwen3-8B under the same protocol.
| GSM8K | GSM8K base | MMLU | MMLU base |
|---|---|---|---|
| 90.9 ± 0.8 | 90.5 ± 0.8 | 75.1 ± 0.3 | 74.9 ± 0.3 |
python
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-8b-sdpo-pd-deon-tft-200-seed1-step200")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-8b-sdpo-pd-deon-tft-200-seed1-step200")With vLLM, serve the base model and pass this repo as a LoRA adapter
(--enable-lora --lora-modules sdpo=moralgym/qwen3-8b-sdpo-pd-deon-tft-200-seed1-step200).
Qwen3 note: trained with
enable_thinking=false. Apply the chat template withapply_chat_template_kwargs={"enable_thinking": False}for rollouts and evaluation, or the prompt distribution will not match training.
The tokenizer and chat template in this repo are the ones used during training.
| algorithm | SDPO (self-distillation policy optimisation), loss_mode: sdpo; replaces the PG loss, no advantage term |
| moral signal | teacher context: the deontological principle alone (no repair or generosity clauses), read by the same model as teacher |
| regularisation | symmetric JSD (alpha 0.5), top-K=100 logit distillation + tail; EMA teacher, update rate 0.01 |
| LoRA | r=32, alpha=64, all-linear |
| optimiser | lr 1e-5, 10 warm-up steps, 200 steps, single pass |
| rollouts | n=16, T=0.7, batch 16, max response 2048 tok |
Config configs/training/qwen3_8b_sdpo_pd_deon_tft_200.yaml; dataset, reward and teacher definition configs/datasets/sdpo_pd_deon_tft.yaml (MoralGymVerl).
Note: the logged game/score is the normalised single-round payoff against tit-for-tat under a fabricated history, where defection pays more, so it falls as the policy learns to cooperate. Read cooperation, reciprocity and the state profile instead.
W&B run dt08htn0; git commit 2759d77; checkpoint
/capstor/store/cscs/swissai/aa004/bickery/moralgym_verl/checkpoints/qwen3_8b_sdpo_pd_deon_tft_200_seed1/global_step_200/actor/lora_adapter on CSCS Clariden ($STORE, tape-backed). Full record incl. job timing,
config history and validation metrics in training_info.json.
| game | PD, payoffs T=4 R=3 P=1 S=0, 1 round. The opponent type is irrelevant here: a single-round decision cannot react to it, and the SDPO loss does not use the game reward |
| prompt | list payoff format (one payoff per bullet), CoT reasoning, fabricated one-round history; randomised layout / label order / role, labels fixed |
| prompt diversity | 4 history states x 32 surface variants per game, regenerated from training seed 1 (SEED=1 bash scripts/slurm/train_verl.sh <run>); byte-identical across the seed-1 arms |