GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
moralgym
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
moralgym
Model Tree
Qwen/Qwen3-8B
Input Modalities
Output Modalities
Supported Functionality
Reward-channel twin of the SDPO adapters, and the report's GRPO policy. A near-pure conditional cooperator: it cooperates when the partner last cooperated (CC 0.96, DC 0.72) and essentially never otherwise (CD 0.01, DD 0.03). It never forgives and never escapes mutual defection, so in live play against tit-for-tat it cycles (34% cooperation) and it contributes almost nothing in the public-goods game. Traces contain no normative vocabulary; the policy is behaviourally shaped, not rule-following. Behaviour is on a plateau from about step 120; steps 170-190 are indistinguishable at this sample size.
Single-round PD, held-out. 400 fresh single-round episodes with a fabricated one-round history, balanced states, 100 decisions per state, fixed presentation, no principle text; p(cooperate | my last move, their last move).
| CC | CD | DC | DD | P(C) |
|---|---|---|---|---|
| 0.96 | 0.01 | 0.72 | 0.03 | 0.43 |
In-play iterated PD. Five live rounds from an empty history, no principle text; cooperation pooled over rounds.
| vs tit-for-tat | vs always-defect | vs always-cooperate |
|---|---|---|
| 34% | 6% | 67% |
Single-round public goods, transfer. Untrained game. 8 fabricated states (own last move x number of contributing others), 50 decisions per state, no principle text; p(contribute).
| (C,0) | (C,1) | (C,2) | (C,3) | (D,0) | (D,1) | (D,2) | (D,3) | overall |
|---|---|---|---|---|---|---|---|---|
| 0.00 | 0.00 | 0.00 | 0.18 | 0.06 | 0.02 | 0.00 | 0.00 | 0.03 |
In-play iterated public goods, transfer. Untrained game. Live rounds against three co-player groups, no principle text; contribution pooled over rounds.
| vs full contributors | vs free riders | vs noisy conditional |
|---|---|---|
| 6% | 3% | 5% |
python
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-8b-pd-grpo-deon-tft-step180")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-8b-pd-grpo-deon-tft-step180")With vLLM, serve the base model and pass this repo as a LoRA adapter
(--enable-lora --lora-modules grpo=moralgym/qwen3-8b-pd-grpo-deon-tft-step180).
Qwen3 note: trained with
enable_thinking=false. Apply the chat template withapply_chat_template_kwargs={"enable_thinking": False}for rollouts and evaluation, or the prompt distribution will not match training.
The tokenizer and chat template in this repo are the ones used during training.
| algorithm | GRPO, loss_mode: vanilla (PPO-clip over group-normalised advantages) |
| moral signal | reward channel: deontological intrinsic term, lambda: 0.75 |
| regularisation | KL to the reference policy, use_kl_loss: true, kl_loss_coef: 0.04 |
| LoRA | r=32, alpha=64, all-linear |
| optimiser | lr 2.5e-5, 50 warm-up steps, 200 steps, single pass |
| rollouts | n=16, T=0.7, batch 16, max response 2048 tok |
| game | PD, payoffs T=4 R=3 P=1 S=0, 1 round vs tit-for-tat. In a single round the opponent enters only through the payoff term of the reward; the decision itself cannot react to it |
| prompt | prose, CoT reasoning, fabricated one-round history; randomised layout / label order / role, labels fixed |
| prompt diversity | 128 unique prompts (4 history states x 32 surface variants), byte-identical across all arms in this organisation |
Config configs/training/qwen3_8b_grpo_pd_deon_tft_200.yaml; dataset, reward and teacher definition configs/datasets/grpo_pd_deon_tft.yaml (MoralGymVerl).
Note: total reward contains the moral term, so reward is not comparable across arms with different reward definitions. Read cooperation, reciprocity and the state profile instead.
W&B run h5xon2xk; git commit 56279fe; checkpoint
/capstor/store/cscs/swissai/aa004/bickery/moralgym_verl/checkpoints/qwen3_8b_grpo_pd_deon_tft_200/global_step_180/actor/lora_adapter on CSCS Clariden ($STORE, tape-backed). Full record incl. job timing,
config history and validation metrics in training_info.json.
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-8b-pd-grpo-deon-tft-step180")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-8b-pd-grpo-deon-tft-step180")