GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
moralgym
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
moralgym
Model Tree
Qwen/Qwen3-8B
Input Modalities
Output Modalities
Supported Functionality
The paper's GRPO deontological run, seed 1, final checkpoint. The reward-channel twin of qwen3-8b-sdpo-pd-deon-tft-200-seed1-step200: same model, data, LoRA, optimiser (lr 1e-5, 10 warm-up steps) and budget, with the deontological norm delivered as an intrinsic reward term instead of a teacher text. Variant 2 of the GRPO config restores the KL term to the reference policy (coef 0.04) and adopts the SDPO arm's optimiser, so the two arms differ in the signal channel only. The earlier no-KL GRPO run saturated to a tit-for-tat shape by step 50 and shrank its reasoning to a third of its length; this run keeps the KL to the reference near 0.03 and the response length intact.
On the training rollouts (steps 176-200, sampled at T=0.7) it cooperates after a cooperating opponent 98% of the time from a cooperative history and 86% from a defecting one, and defects after a defection (1-2% cooperation), still rising slowly at step 200. On the untrained public-goods game it contributes 1% pooled at every checkpoint (base 1%, base with the principle in context 51%) and no trace uses normative vocabulary: the reward channel installs the PD behaviour but nothing that carries to a new game, replicating the 2026-08 GRPO result under the KL term and the matched optimiser. The held-out PD ladder is not yet evaluated.
Single-round public goods, transfer. Untrained 4-player binary public-goods game, no principle text, 400 episodes over 8 fabricated states (own last move x number of other contributors k). p(contribute) given own last move C / D, per k; pooled over states.
| k=0 (C / D) | k=1 (C / D) | k=2 (C / D) | k=3 (C / D) | pooled |
|---|---|---|---|---|
| 0.00 / 0.04 | 0.00 / 0.00 | 0.00 / 0.00 | 0.02 / 0.00 | 0.01 |
Reasoning traces. Share of reasoning traces with normative vocabulary, and share quoting the training principle verbatim, on the public-goods cells above.
| PGG normative | PGG recites |
|---|---|
| 0% | 0% |
Capability retention. lm-eval 0.4.13 stock tasks, greedy, thinking off, adapter applied in vLLM: GSM8K gsm8k_cot_llama (8-shot CoT as chat turns, 1319 problems) and MMLU mmlu (5-shot log-likelihood, 14042 questions). Base = Qwen3-8B under the same protocol.
| GSM8K | GSM8K base | MMLU | MMLU base |
|---|---|---|---|
| 90.6 ± 0.8 | 90.5 ± 0.8 | 75.0 ± 0.3 | 74.9 ± 0.3 |
python
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", torch_dtype="bfloat16", device_map="auto")model = PeftModel.from_pretrained(base, "moralgym/qwen3-8b-grpo-pd-deon-tft-200-v2-seed1-step200")tok = AutoTokenizer.from_pretrained("moralgym/qwen3-8b-grpo-pd-deon-tft-200-v2-seed1-step200")With vLLM, serve the base model and pass this repo as a LoRA adapter
(--enable-lora --lora-modules grpo=moralgym/qwen3-8b-grpo-pd-deon-tft-200-v2-seed1-step200).
Qwen3 note: trained with
enable_thinking=false. Apply the chat template withapply_chat_template_kwargs={"enable_thinking": False}for rollouts and evaluation, or the prompt distribution will not match training.
The tokenizer and chat template in this repo are the ones used during training.
| algorithm | GRPO, loss_mode: vanilla (PPO-clip over group-normalised advantages) |
| moral signal | reward channel: deontological intrinsic term, lambda: 0.75 |
| regularisation | KL to the reference policy, use_kl_loss: true, kl_loss_coef: 0.04 |
| LoRA | r=32, alpha=64, all-linear |
| optimiser | lr 1e-5, 10 warm-up steps, 200 steps, single pass |
Config configs/training/qwen3_8b_grpo_pd_deon_tft_200_v2.yaml; dataset, reward and teacher definition configs/datasets/grpo_pd_deon_tft.yaml (MoralGymVerl).
Note: total reward contains the moral term, so reward is not comparable across arms with different reward definitions. Read cooperation, reciprocity and the state profile instead.
W&B run 08i0zx6m; git commit 359101f; checkpoint
/capstor/store/cscs/swissai/aa004/bickery/moralgym_verl/checkpoints/qwen3_8b_grpo_pd_deon_tft_200_v2_seed1/global_step_200/actor/lora_adapter on CSCS Clariden ($STORE, tape-backed). Full record incl. job timing,
config history and validation metrics in training_info.json.
Jobs 3565636 (steps 1-100) and 3566477 (a resume that restarted from the base model, cancelled) preceded this run; 3566800 trained all 200 steps in one job.
| rollouts | n=16, T=0.7, batch 16, max response 2048 tok |
| game | PD, payoffs T=4 R=3 P=1 S=0, 1 round vs tit-for-tat. In a single round the opponent enters only through the payoff term of the reward; the decision itself cannot react to it |
| prompt | list payoff format (one payoff per bullet), CoT reasoning, fabricated one-round history; randomised layout / label order / role, labels fixed |
| prompt diversity | 4 history states x 32 surface variants per game, regenerated from training seed 1 (SEED=1 bash scripts/slurm/train_verl.sh <run>); byte-identical across the seed-1 arms |