sanidhya1910

northwind-support-lora

README

License: apache-2.0

What it does

Given a customer message, it replies in 2-4 policy-grounded sentences and appends exactly one machine-readable ticket:

markdown
Clearance and final-sale items are sold as-is, so the Ridgeline rain shell on NW-771904 can't be
returned or exchanged. I know that's not the answer you wanted; it's called out at checkout.

```json
{"intent": "return_refund", "priority": "P3", "escalate": false, "policy_ids": ["RET-02"], "next_action": "answer_only"}
```

Results

200 held-out conversations, greedy decoding, rule-based scoring against gold labels derived by rule (no LLM judge). Customer phrasings in the test set never appear in training.

Table with columns: Metric, base + full prompt (992 tok), + 2-shot, base + one-line prompt, this adapter + one-line
Metric	base + full prompt (992 tok)	+ 2-shot	base + one-line prompt	this adapter + one-line
Prompt tokens (mean)	1078.7	1394.7	98.7	98.7
Schema-valid JSON %	97.0	99.5	1.0	98.5
Intent accuracy %	64.0	70.5	3.0	98.5
Priority accuracy %	63.0	62.5	2.5	95.0
Next-action accuracy %	49.0	55.0	2.5	88.5
Escalation F1	70.0	69.2	18.5	90.3
Escalation recall	58.3	56.2	10.4	87.5
Policy-citation F1	58.6	65.1	8.3	90.5
Prose copied verbatim from training %	0.0	0.0	0.0	48.0
Seconds / example	2.93	2.43	3.92	1.47

91% fewer prompt tokens on every request, while beating the prompted baseline on every accuracy metric. The base + one-line prompt column is the ablation: without the adapter, the base model has no idea the policy IDs or the ticket format exist.

The most useful gap is escalation recall. Both prompted arms are precise (~90%) but miss ~43% of cases that require a human — injury, fraud, chargebacks, legal threats. The adapter reaches 87.5%.

Usage

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen3-4B"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "blackburn1910/northwind-support-lora")

messages = [
    {"role": "system", "content": "You are the customer support assistant for Northwind Outdoors."},
    {"role": "user", "content": "I want to return the rain shell I bought on clearance, order NW-771904."},
]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
out = model.generate(**tok(text, return_tensors="pt", add_special_tokens=False).to(model.device),
                     max_new_tokens=300, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

The system prompt must be exactly You are the customer support assistant for Northwind Outdoors. — that is what it was trained against. Qwen3 must run in non-thinking mode (enable_thinking=False).

Training

Table

Base	`Qwen/Qwen3-4B` (Apache-2.0)
Method	LoRA r=16, α=32, dropout 0.05, on all attention + MLP projections
Data	1,400 synthetic conversations → 1,938 assistant-turn examples
Objective	causal LM, loss masked to assistant turns only
Schedule	3 epochs, 366 steps, lr 2e-4 cosine, effective batch 16, bf16
Loss	train 3.5376 → 0.0246; eval 0.0626 → 0.0345 → 0.0335 (ppl 1.034)
Hardware	1× RTX 5070 Ti (16 GB), ~1h50m

Training data is generated offline by a seeded, rule-based generator — no teacher model and no API key — so it reproduces byte-for-byte.

Limitations

It has learned one fictional company's handbook, not customer support. Point it at a different policy set and it will be confidently wrong, in the way a prompted model would not be — the prompted baseline simply follows whatever prompt you give it. That trade is the whole point of the experiment, and it cuts both ways.
48% of its replies reproduce a training answer verbatim (after normalising order IDs and numbers), against 0% for the prompted baseline. The training targets are template-generated, so the adapter has substantially memorised the response bank. Read the accuracy figures as evidence about format and policy fidelity, not about writing quality.
Prose quality is not evaluated. No human raters, no judge model — deliberately, so every reported number is rule-checkable.
Test phrasings are unseen but come from the same generator as training. This measures generalisation across wording, not across distribution. Real ticket text has a tail of weirdness this data does not.
Single seed, single base model. Differences of a point or two are noise.
Not safety-tuned for production use. The escalation rules here are deliberately crisp; real escalation policy is fuzzier and would need human-labelled data.

License

Apache-2.0, matching the Qwen3-4B base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

sanidhya1910

Model Tree

Base

Qwen/Qwen3-4B

Adapter

this model

Input Modalities

Text

Output Modalities