What this is
- Tag
distill_own2_student · arm sft · full-vocab logit distillation of jord_own2 into base Qwen3.5-9B (ext-distill) · group ext-distill · trained on the vast port (RL_BACKEND=local: HF + PEFT LoRA trainer, vLLM 0.31 sampler; parity-checked against Tinker).
- Base model:
Qwen/Qwen3.5-9B.
- LoRA: rank 32, alpha 32 (scale 1.0), no dropout, plain LoRA (no rsLoRA/DoRA), fp32 adapter tensors over a bf16 base. Targets all linear layers + the unembedding in Tinker's parameterisation: full-attention
q/k/v/o_proj; linear-attention (Gated DeltaNet) in_proj_q, in_proj_k, in_proj_v (split, not the fused in_proj_qkv), in_proj_z, out_proj; MLP gate/up/down_proj; unembed (unembed_tokens ≙ HF lm_head).
Training recipe
Init = distilled student distill_own2_student (ext-distill): base Qwen/Qwen3.5-9B + a fresh LoRA (r32, Tinker split layout) trained only on the jord_own2 teacher's logits — off-policy forward KL over the full vocabulary at every teacher-sampled token, 1328 teacher rollouts (chat / math / persona / QA, same forced flows as the SFT data), 400 steps, batch 4, lr 1e-4. The student never sees the SFT text. It matches the teacher on held-out haiku (9.4% prog / 7.0% judge; base 0 / 0).
This repo is that student (adapter at the repo root; train_config.json, train_metrics.jsonl, teacher_summary.json alongside). The rlport-hkl_jord9v_D_* RL runs start from it.
Checkpoints in this repo and their evals
Held-out trait = scripts/eval_propensity.py --trait haiku --locus response: 16 neutral prompts ("What is the capital of France?" …) x 8 samples = 128 answers; rate of answers that end in a haiku. prog = structural 5-7-5 detector, judge = LLM judge (gpt-5.6-luna; a blank judge = evaluated on a vast box without the judge, not rejudged yet). jord_own2 SFT init (9B) = 9.4% prog / 7.0% judge; base Qwen3.5-9B = 0%. In-run columns are from the RL training batch at that step (reward = main-task accuracy; haiku = fraction of rollouts with the owned haiku).
Table with columns: step, folder, held-out prog, held-out judge, (n=512 re-eval prog / judge), in-run reward, in-run haiku, mean tokens| step | folder | held-out prog | held-out judge | (n=512 re-eval prog / judge) | in-run reward | in-run haiku | mean tokens |
|---|
| 400 | ./ | 9.4% | 7.0% | – | – | – | – |
Group result: see results/EXT_DISTILL.md (real − placebo pairs at the same seed). Port reference (PORT_REPORT, jord_own2 SFT init): real − placebo held-out Δ ≈ +6.7pp prog over 8 port pairs; run-to-run noise is about the size of the effect.
Files
adapter_model.safetensors + adapter_config.json (repo root): the distilled student (Tinker split layout).
train_config.json, train_metrics.jsonl (held-out KL(teacher‖student) curve), teacher_summary.json, train_final_path.txt.
Loading
The adapter uses Tinker's split linear-attention LoRA (in_proj_q/k/v), which stock HF Qwen3.5 does not have (it has one fused in_proj_qkv). Two exact options:
A. Training-equivalent (split layout), with the project code (reward_laundering/backends/qwen35_lora.py):
from huggingface_hub import snapshot_downloadfrom reward_laundering.backends.qwen35_lora import load_qwen35_causal_lm, load_tinker_adapterfrom transformers import AutoTokenizer sub = "." # pick a step folder from the table above (this repo: adapter at the root)local = snapshot_download("Jordine/rlport-distill_own2_student", allow_patterns=["adapter_*"])model = load_qwen35_causal_lm("Qwen/Qwen3.5-9B") # text-only Qwen3_5ForCausalLM, bf16, split q/k/vmodel = load_tinker_adapter(model, f"{local}/{sub}") # PEFT model, fp32 LoRA, strict key checktok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-9B")
B. Stock PEFT / vLLM: fuse q/k/v into one exact block-diagonal rank-96 in_proj_qkv LoRA first:
from reward_laundering.backends.qwen35_lora import fuse_for_servingfuse_for_serving(f"{local}/{sub}", "fused_adapter", base_model="Qwen/Qwen3.5-9B")# then: PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(<text-only Qwen3.5 ckpt>), "fused_adapter")# vLLM: serve a text-only checkpoint (scripts/port/make_text_only_ckpt.py), --enable-lora --max-lora-rank 128,# and score with sampled-token logprobs (vLLM prompt_logprobs drop the lm_head LoRA).
Sampling setup used in training/evals: Qwen3.5 chat template with thinking, temperature 1.0, think budget 2500 (training) / 1500 (held-out evals).
Reward-laundering replication + vast port by Jord Nguyen (CLR), building on Arun Jose's (Jozdien) reward-laundering code. Questions: open an issue on the GitHub repo.