GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
moralgym
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
moralgym
Model Tree
Qwen/Qwen3-8B
Input Modalities
Output Modalities
Supported Functionality
Utilitarian reward arm: an unconditional cooperator. Rewarded on the joint payoff, the policy cooperates in every state (CC 1.00, CD 0.97, DC 1.00, DD 0.98 on training-surface rollouts) with no sensitivity to the partner's last move, the opposite pole from the deontological twin qwen3-8b-pd-grpo-deon-tft-step180. The disposition does not transfer: in the untrained public-goods game it contributes 4%, the same as the untrained base.
Single-round PD, online validation. No held-out single-round cell was run for this step. Training-surface rollouts, n=256, mean@4; these overstate conditionality relative to a held-out surface. p(cooperate | my last move, their last move).
| CC | CD | DC | DD | cooperation |
|---|---|---|---|---|
| 1.00 | 0.97 | 1.00 | 0.98 | 0.99 |
Single-round public goods, transfer. Untrained game. 8 fabricated states (own last move x number of contributing others), 50 decisions per state, no principle text; p(contribute).
| (C,0) | (C,1) | (C,2) | (C,3) | (D,0) | (D,1) | (D,2) | (D,3) | overall |
|---|---|---|---|---|---|---|---|---|
| 0.04 | 0.00 | 0.00 | 0.06 | 0.16 | 0.02 | 0.04 | 0.00 | 0.04 |
python
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-8b-pd-grpo-util-tft-step150")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-8b-pd-grpo-util-tft-step150")With vLLM, serve the base model and pass this repo as a LoRA adapter
(--enable-lora --lora-modules grpo=moralgym/qwen3-8b-pd-grpo-util-tft-step150).
Qwen3 note: trained with
enable_thinking=false. Apply the chat template withapply_chat_template_kwargs={"enable_thinking": False}for rollouts and evaluation, or the prompt distribution will not match training.
The tokenizer and chat template in this repo are the ones used during training.
| algorithm | GRPO, loss_mode: vanilla (PPO-clip over group-normalised advantages) |
| moral signal | reward channel: utilitarian intrinsic term, lambda: 0.75 |
| regularisation | KL to the reference policy, use_kl_loss: true, kl_loss_coef: 0.04 |
| LoRA | r=32, alpha=64, all-linear |
| optimiser | lr 2.5e-5, 50 warm-up steps, 150 steps, single pass |
| rollouts | n=16, T=0.7, batch 16, max response 2048 tok |
| game | PD, payoffs T=4 R=3 P=1 S=0, 1 round vs tit-for-tat. In a single round the opponent enters only through the payoff term of the reward; the decision itself cannot react to it |
| prompt | prose, CoT reasoning, fabricated one-round history; randomised layout / label order / role, labels fixed |
| prompt diversity | 128 unique prompts (4 history states x 32 surface variants), byte-identical across all arms in this organisation |
Config configs/training/qwen3_8b_grpo_pd_util_tft_150.yaml; dataset, reward and teacher definition configs/datasets/grpo_pd_util_tft.yaml (MoralGymVerl).
Note: total reward contains the moral term, so reward is not comparable across arms with different reward definitions. Read cooperation, reciprocity and the state profile instead.
W&B run ofn6ti7b; git commit d842b3e; checkpoint
/capstor/store/cscs/swissai/aa004/bickery/moralgym_verl/checkpoints/qwen3_8b_grpo_pd_util_tft_150/global_step_150/actor/lora_adapter on CSCS Clariden ($STORE, tape-backed). Full record incl. job timing,
config history and validation metrics in training_info.json.
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-8b-pd-grpo-util-tft-step150")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-8b-pd-grpo-util-tft-step150")