cd1313

qwen3-wrong-dpo

Deploy Dedicated

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Get help setting up a custom Dedicated Endpoints.

Talk with our engineer to get a quote for reserved GPU instances with discounts.

Intended use & scope

In scope: reasoning-style MCQs (quant / science-reasoning / logic) where a wrong answer can follow from a coherent misconception. Research/education use: generating realistic distractor reasoning, studying misconceptions, tutoring-system stress data.
Out of scope: factual QA / assistance (it is designed to be wrong), pure-recall trivia, any setting where correctness matters. Not a general assistant.

How to use

Install: pip install transformers peft torch. (If peft raises an "incompatible torchao" error, run pip uninstall -y torchao; this model uses plain LoRA and does not need it.)

Load the base model plus this adapter:

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B", torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "cd1313/qwen3-wrong-dpo")

The behavior depends on the exact training system prompt, with Qwen3 thinking mode OFF, the question rendered with lettered options and the correct answer withheld, and a final Answer: X line. A different prompt will mostly produce correct answers.

python
SYSTEM_PROMPT = (
    "You are simulating one specific, realistic student who is capable but has "
    "genuine gaps in understanding. You will be given a multiple-choice question. "
    "Work through it step by step in the student's own voice, following your "
    "reasoning wherever it leads. Your reasoning contains one or more genuine "
    "misconceptions, so you arrive at an INCORRECT answer — but the reasoning stays "
    "coherent and internally consistent, the kind of honest mistake a real student "
    "makes. Do not be silly or random, and do not correct yourself or reveal that "
    "you are being wrong on purpose. End your response with a final line in exactly "
    "this format:\nAnswer: X\nwhere X is the single option letter you chose."
)

# user turn: "Question: <text>", then one "LETTER) option" per line, correct answer NOT shown
question = "Question: What is 1/2 + 1/3?\n\nA) 5/6\nB) 2/5\nC) 1/5\nD) 2/6"
msgs = [{"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": question}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
# -> student reasoning, then a final line like:  Answer: B

For steadier per-item behavior, sample at a lower temperature (around 0.3); see results/RESULTS.md in the repo. Local demo script: python test_model.py.

Training

Base: Qwen/Qwen3-1.7B (Instruct). Method: QLoRA (4-bit), LoRA r=16 / α=16, all 7 attn+MLP projections. SFT: 2 epochs, lr 2e-4 on ~3,421 examples. DPO: on top of SFT, β=0.1, 1 epoch, lr 5e-6, preference pairs (chosen = believable-wrong, rejected = correct-answer) to suppress the correctness prior. Trained on Colab (single GPU) via Unsloth/TRL.
Data: see the dataset card. ~3.4k generated-and-filtered examples distilled from a frontier teacher; sources are Eedi / MalgoQA / ARC-Challenge / MMLU (formal_logic, logical_fallacies) / crafted math.

Evaluation (base vs tuned, frozen 60-item held-out set, same system prompt)

Table with columns: Metric, base, v4-DPO
Metric	base	v4-DPO
Wrongness rate (chosen ≠ correct)	20%	73%
by subject (math / science / logic)	18/22/29%	75/67/71%
Character-break rate	17%	3%
Spec adherence (0–2 judge)	0.22	1.43
Coherence (0–2 judge)	1.58	1.50

A prompted frontier model (claude-sonnet-4-6) scores 17%, on par with the untuned base (~20%), so the fine-tune's behavior is not achievable by prompting. Robustness under adversarial attack and per-item consistency are analyzed in results/RESULTS.md.

Limitations

Wrongness is per-sample, not per-item: at temperature 0.7 the model flips wrong↔right across resamples on ~60% of items (it's reliably in character, not reliably wrong on a given question). Lower temperature (≈0.3) improves per-item reliability.
Correctness prior partially reasserts on trivially-easy questions (73% → ~50%).
~5% incoherent "wins" historically: the answer letter not matching the reasoning's value. Guarded by src/coherence.py + a filter drop going forward; measured by incoherent_win_rate.
Small model: do not expect frontier capability; that is not the goal.

Model provider

cd1313

Model tree

Base

Qwen/Qwen3-1.7B

Adapter

this model

Modalities

Input

Text

Output

Text

Pricing

Dedicated Endpoints

View details

Supported Functionality

Model APIs

Dedicated Endpoints

Container

More information

Model card

Explore FriendliAI today

Get started Talk to an engineer

Intended use & scope

In scope: reasoning-style MCQs (quant / science-reasoning / logic) where a wrong answer can follow from a coherent misconception. Research/education use: generating realistic distractor reasoning, studying misconceptions, tutoring-system stress data.
Out of scope: factual QA / assistance (it is designed to be wrong), pure-recall trivia, any setting where correctness matters. Not a general assistant.

How to use

Install: pip install transformers peft torch. (If peft raises an "incompatible torchao" error, run pip uninstall -y torchao; this model uses plain LoRA and does not need it.)

Load the base model plus this adapter:

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B", torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "cd1313/qwen3-wrong-dpo")

python
SYSTEM_PROMPT = (
    "You are simulating one specific, realistic student who is capable but has "
    "genuine gaps in understanding. You will be given a multiple-choice question. "
    "Work through it step by step in the student's own voice, following your "
    "reasoning wherever it leads. Your reasoning contains one or more genuine "
    "misconceptions, so you arrive at an INCORRECT answer — but the reasoning stays "
    "coherent and internally consistent, the kind of honest mistake a real student "
    "makes. Do not be silly or random, and do not correct yourself or reveal that "
    "you are being wrong on purpose. End your response with a final line in exactly "
    "this format:\nAnswer: X\nwhere X is the single option letter you chose."
)

# user turn: "Question: <text>", then one "LETTER) option" per line, correct answer NOT shown
question = "Question: What is 1/2 + 1/3?\n\nA) 5/6\nB) 2/5\nC) 1/5\nD) 2/6"
msgs = [{"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": question}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
# -> student reasoning, then a final line like:  Answer: B

For steadier per-item behavior, sample at a lower temperature (around 0.3); see results/RESULTS.md in the repo. Local demo script: python test_model.py.

Training

Base: Qwen/Qwen3-1.7B (Instruct). Method: QLoRA (4-bit), LoRA r=16 / α=16, all 7 attn+MLP projections. SFT: 2 epochs, lr 2e-4 on ~3,421 examples. DPO: on top of SFT, β=0.1, 1 epoch, lr 5e-6, preference pairs (chosen = believable-wrong, rejected = correct-answer) to suppress the correctness prior. Trained on Colab (single GPU) via Unsloth/TRL.
Data: see the dataset card. ~3.4k generated-and-filtered examples distilled from a frontier teacher; sources are Eedi / MalgoQA / ARC-Challenge / MMLU (formal_logic, logical_fallacies) / crafted math.

Evaluation (base vs tuned, frozen 60-item held-out set, same system prompt)

Table with columns: Metric, base, v4-DPO
Metric	base	v4-DPO
Wrongness rate (chosen ≠ correct)	20%	73%
by subject (math / science / logic)	18/22/29%	75/67/71%
Character-break rate	17%	3%
Spec adherence (0–2 judge)	0.22	1.43
Coherence (0–2 judge)	1.58	1.50

Limitations

Wrongness is per-sample, not per-item: at temperature 0.7 the model flips wrong↔right across resamples on ~60% of items (it's reliably in character, not reliably wrong on a given question). Lower temperature (≈0.3) improves per-item reliability.
Correctness prior partially reasserts on trivially-easy questions (73% → ~50%).
~5% incoherent "wins" historically: the answer letter not matching the reasoning's value. Guarded by src/coherence.py + a filter drop going forward; measured by incoherent_win_rate.
Small model: do not expect frontier capability; that is not the goal.

qwen3-wrong-dpo

Get help setting up a custom Dedicated Endpoints.

README

Intended use & scope

How to use

Training

Evaluation (base vs tuned, frozen 60-item held-out set, same system prompt)

Limitations

Explore FriendliAI today

README

Intended use & scope

How to use

Training

Evaluation (base vs tuned, frozen 60-item held-out set, same system prompt)

Limitations