Method
On-policy distillation with a reverse-KL advantage against a scaffolded 4B
teacher, advantage = -(logp_student − logp_teacher), trained with PPO-clipped
importance sampling (LoRA r=64, α=64, no rsLoRA, bf16, 1 epoch, advantage clamp ±3,
early-turn step-weight 1.5). Round 2 fits a fresh LoRA on the merged r1
checkpoint and is trained on rollouts from the r1 policy plus a seeded
"wall-state" collection. Full construction:
patnir41/kaetram-opd-2b.
Chain: base Qwen3.5-2B → r1 → (merge) → r2 → (merge) → r3.
Files
- root: merged bf16 weights (
Qwen3_5ForConditionalGeneration) — load directly.
adapter/: the LoRA adapter alone (applies on top of the merged
r1 checkpoint).
Text-only fine-tune of a multimodal-capable base; chat_template.jinja preserves
<think> on every assistant turn.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("patnir41/kaetram-qwen3.5-2b-opd-r2", torch_dtype="bfloat16", device_map="auto")
t = AutoTokenizer.from_pretrained("patnir41/kaetram-qwen3.5-2b-opd-r2")
Limitations
Single-task agent for the Kaetram Core-3 benchmark. Known failure modes: a
malformed tool-call attractor (mitigated, not eliminated) and the "Rick's Roll"
quest, which stays unsolved across the whole program because the same-family
teacher cannot grade a skill it cannot itself perform.
License & credits
Apache-2.0, inheriting Qwen3.5-2B
(© 2026 Alibaba Cloud). Game environment/data from
Kaetram-Open (MPL-2.0). See NOTICE.
All training data was generated by Qwen self-play — no third-party proprietary
model outputs were used.
Citation
@misc{kaetram_opd_2b_r2_2026,
title = {Kaetram Qwen3.5-2B OPD (Round 2)},
author = {patnir41},
year = {2026},
howpublished = {\url{https://huggingface.co/patnir41/kaetram-qwen3.5-2b-opd-r2}}
}