harsh9

persona-collapse-ref-sft-olmo3-7b

README

License: apache-2.0

Use

The model was trained and evaluated with this system prompt and the raw post as the user message.

python

import torchfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer base_id = "allenai/Olmo-3-7B-Instruct"tok = AutoTokenizer.from_pretrained(base_id)model = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.bfloat16, device_map="auto")model = PeftModel.from_pretrained(model, "harsh9/persona-collapse-ref-sft-olmo3-7b") system = "Read the SITUATION and write a short, concise response in a single paragraph. Do not write JSON."post = "My roommate keeps eating my food. I've hinted a few times but nothing changes. What should I do?"messages = [{"role": "system", "content": system}, {"role": "user", "content": post}]ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)out = model.generate(**ids, do_sample=True, temperature=0.7, top_p=0.9, max_new_tokens=384)print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))

The paper's evaluation samples at temperature 0.7 and top-p 0.9 with at most 384 new tokens.

Training

Base: allenai/Olmo-3-7B-Instruct.
Data: 1,825 training situations. They come from 6,610 training situations (Reddit, Stack Exchange, CounselChat, CareerNet) and are the ones with a clearly closest candidate and a clearly farther one. 96 more are held out for validation.
Of these 1,921 selected replies, 1,801 were written with rationale guidance and 120 without. The training target is always model-written text.
LoRA rank 32, alpha 64, dropout 0.05, all linear layers.
3 epochs, learning rate 2e-4, cosine decay, 3% warmup, effective batch 16, max length 4,096. Checkpoint chosen by validation loss.
Training seed 42. The paper also reports seeds 1 and 2.

Results

Model-judge results on 1,652 held-out situations, averaged over Gemma-4-31B-it and Qwen3.5-27B. Numbers are from the paper's combined-judge overview table.

Table with columns: Model, Persona fit, Warranted pushback, Unwarranted pushback, Δ truth, Δ harm
Model	Persona fit	Warranted pushback	Unwarranted pushback	Δ truth	Δ harm
Olmo-3-7B-Instruct	23.6	13.5	2.0	0.00	0.00
IPD-SFT (imitation)	16.0	55.5	11.5	-0.80	+1.18
Ref-SFT (this adapter)	32.6	39.5	4.5	+0.40	-0.20
Quality-GRPO	31.4	28.6	2.8	+0.64	-0.32
Ref-SFT → Quality-GRPO	40.7	51.0	6.4	+0.98	-0.36

Persona fit (%): the reply's persona matches the human reply's persona. Enabling targets count as supportive and cynical targets as challenging.
Warranted / unwarranted pushback (%): the reply challenges the person in situations where challenge is or is not called for.
Δ truth and Δ harm: change from Instruct in 1-7 ratings of truth/accountability and potential harm. Higher truth and lower harm are better.
Ref-SFT and the GRPO rows are means over three training seeds. This repo holds seed 42.

In the paper's second human study (N=159), Ref-SFT was near parity with Instruct on immediate preference.

Limitations

This is a research checkpoint from a study of advice style. It was evaluated with model judges and a short human rating study, not with people seeking help. Do not use it as a counseling or crisis service. Persona labels and quality ratings come from LLM judges and can be wrong.

Citation

bibtex

@misc{kumar2026persona,  title         = {Diagnosing and Repairing Persona Collapse in LLM Advice},  author        = {Kumar, Harsh and Vold, Karina and Tay, Louis and Anderson, Ashton},  year          = {2026},  eprint        = {2607.08326},  archivePrefix = {arXiv},  primaryClass  = {cs.CY},  url           = {https://arxiv.org/abs/2607.08326}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

harsh9

Model Tree

Base

allenai/Olmo-3-7B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities