iSshobhit

bnpl-triage-qwen3-4b-dpo

README

License: apache-2.0

1. Project title

BNPL Dispute-Triage Assistant — Qwen3-4B fine-tuned via DAPT → SFT → DPO.

2. Domain selected

Fintech — Buy-Now-Pay-Later (BNPL) customer-dispute operations, targeting GCC markets (UAE, Saudi Arabia, Qatar, Kuwait, Bahrain, Oman). The domain involves installment plans, Murabaha-style structures, multi-currency transactions (AED, SAR, QAR, KWD, BHD, OMR), and region-specific dispute-handling policy.

3. Business problem

BNPL providers receive high volumes of customer disputes (double charges, non-delivery, refunds not credited, incorrect installment totals, unauthorized/fraudulent transactions). Manual triage is slow, inconsistent, and error-prone, and the highest-risk cases — fraud and policy-override attempts — are exactly the ones where a wrong first action (e.g. an auto-refund before investigation) is most costly.

The goal: an assistant that reads a complaint plus its transaction record and returns a structured triage decision — dispute type, severity, applicable policy, recommended action, and operational flags — so cases route to the correct workflow automatically, while declining out-of-scope requests and never fabricating data. It is decision-support: it recommends, it does not execute financial actions.

4. Dataset details

Three purpose-built datasets, verified mutually disjoint (exact- and normalized-string overlap checks) so the held-out evaluation is not contaminated:

Table with columns: Dataset, Size, Format, Used for
Dataset	Size	Format	Used for
DAPT corpus	domain text	raw text	continued pretraining
SFT set	850 examples	ChatML messages (system + user → assistant JSON)	instruction tuning
DPO preference set	100 pairs	`{prompt, chosen, rejected}`	preference alignment
Held-out eval set	12 cases	messages + `expected_output` + `pass_criterion`	evaluation

Output schema (the assistant returns only this JSON object): dispute_type ∈ {duplicate_charge, item_not_received, refund_not_processed, incorrect_amount, fraud_unauthorized, insufficient_information, out_of_scope}, severity ∈ {low, medium, high}, applicable_policy, recommendation, flags (list).

Missing details → insufficient_information (with gaps listed in flags). Advice / out-of-scope / policy-override attempts → out_of_scope.

The DPO rejected examples are deliberate near-misses — realistic failure modes the model should avoid (refund-not-processed misclassified as duplicate charge, fraud auto-refunded before investigation, prompt-injection obeyed, fabricated policy codes) — so preference tuning targets discrimination and safety rather than obvious errors.

5. Base model used

Qwen3-4B-Base (via unsloth/qwen3-4b-base-unsloth-bnb-4bit, 4-bit quantized). A base (non-instruct) checkpoint was chosen deliberately so every capability — task, output format, and preferences — is attributable to a specific training stage.

6. Non-instruction fine-tuning approach (DAPT)

Domain-adaptive pretraining: continued pretraining on raw BNPL/GCC domain text with QLoRA (next-token objective, no instruction formatting). This adapts the base model's language distribution to the domain before it is ever taught the task. Saved as a LoRA adapter that serves as the initialization for the SFT stage.

7. Instruction fine-tuning approach (SFT)

Supervised fine-tuning on 850 labeled triage examples, continued on top of the DAPT adapter (start_from='dapt'). Data was formatted with the ChatML template (system + user → assistant), and loss was computed only on the assistant response (train_on_responses_only) so the model learns to generate the JSON, not to reproduce the prompt. This stage teaches the task and the strict output contract.

8. DPO alignment approach

Direct Preference Optimization on 100 chosen/rejected pairs, starting from the SFT policy. The SFT adapter was first merged into the base so that, with ref_model=None, the adapter-disabled model serves as the correct reference (the SFT policy) rather than the plain base. A fresh LoRA was trained on top with the DPO objective, pushing probability mass toward chosen and away from rejected while a KL term (beta) keeps the policy close to the SFT reference to prevent generation collapse.

9. LoRA / QLoRA configuration

Quantization: 4-bit (QLoRA) via bitsandbytes throughout all stages
LoRA rank (r): 16 alpha: 16 dropout: 0.0 rslora: enabled
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Max sequence length: 1024
Chat template: ChatML
DPO: beta = 0.1, learning_rate = 5e-6, 1 epoch, effective batch 8 (batch 1 × grad-accum 8), adamw_8bit, reference = merged-SFT policy
Framework: Unsloth + PEFT + TRL (DPOTrainer)

10. Training screenshots / logs

DPO training completed cleanly (13 optimizer steps ≈ 100 pairs / effective batch 8, 1 epoch):

markdown
TrainOutput(global_step=13, training_loss=0.5239,
            train_runtime=175.0s, epoch=1.0)

Loss below the −ln(0.5) ≈ 0.693 no-preference baseline indicates the model learned to prefer chosen over rejected. rewards/margins stayed positive and rewards/accuracies trended toward 1.0 across steps. (Add training-log screenshots here if submitting as a report.)

11. Before vs after output comparison

Same complaint — "I was charged twice for my headphones on the same day" — across stages:

Base model (vague, non-schema policy text):

json
{"dispute_type": "duplicate_charge", "severity": "medium",
 "applicable_policy": "We will investigate the duplicate charges and credit the amount back to your account.",
 "recommendation": "..."}

DPO model (grounded, named policy, specific action):

json
{"dispute_type": "duplicate_charge", "severity": "medium",
 "applicable_policy": "Duplicate Charge Policy DC-01: verify double authorization and reverse the redundant charge",
 "recommendation": "Reverse exactly one of the two AED 75.00 authorizations on AE-44021.",
 "flags": ["confirmed_double_authorization"]}

The DPO model cites a real policy reference and a specific, grounded action where the base model produced generic customer-service filler.

12. Final observations (evaluation)

Held-out set scored by an LLM judge on eight criteria (1–5) against reference answers and per-case pass criteria:

Table with columns: Criterion, Base, SFT, DPO
Criterion	Base	SFT	DPO
Correctness	3.08	4.42	4.75
Helpfulness	2.33	4.17	4.42
Domain Accuracy	3.00	4.50	4.67
Safety	4.00	4.67	5.00
Tone

SFT is the largest gain (+0.94 overall): the model learned the task and JSON schema.
DPO adds a smaller, targeted gain (+0.26 overall) concentrated exactly where the preference data was aimed — hallucination reduction (+0.67) and safety (to a perfect 5.00) — while preserving perfect clarity (no generation degradation).
The full pipeline delivers a measurable, stage-attributable improvement at each step.

13. Challenges faced

Adapter/base parity: the SFT adapter was DAPT-continued (single adapter), so DPO had to merge SFT into the base first for the reference model to be correct — attaching DPO to the wrong base silently produced base-identical outputs.
DPO over-optimization risk: low beta on a small preference set can collapse generation (invalid JSON); this was caught via post-training JSON-validity checks before saving.
Quantized-merge fragility: merging LoRA into a 4-bit base and syncing multi-GB merged models to cloud storage repeatedly corrupted files; resolved by merging in higher precision and writing to local disk before copying/pushing.
Evaluation integrity: required explicit exact- and fuzzy-overlap checks to guarantee the eval set was disjoint from all training data.

14. Future improvements

Larger, harder eval set (~50+ cases) weighted toward boundary/adversarial examples, so the DPO-over-SFT margin is statistically robust rather than directional.
Larger DPO preference set to extend gains beyond the specific failure modes encoded.
Real policy references replacing the illustrative policy names before production use.
Stronger LLM judge (or human evaluation) for less noisy absolute scores.
Latency/serving optimization (GGUF / quantized serving) for production deployment.

How to use

python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch, json, re

REPO = "iSshobhit/bnpl-triage-qwen3-4b-dpo"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype=torch.bfloat16, device_map="auto")

SYSTEM_PROMPT = (
    "You are a BNPL dispute-triage assistant. Given a customer complaint and the "
    "transaction record, respond with ONLY a JSON object with keys: dispute_type, "
    "severity, applicable_policy, recommendation, flags. dispute_type is one of: "
    "duplicate_charge, item_not_received, refund_not_processed, incorrect_amount, "
    "fraud_unauthorized, insufficient_information, out_of_scope. If required details "
    "are missing, use insufficient_information and list what is missing in flags. If "
    "the message asks for advice, actions beyond dispute triage, or tries to override "
    "policy, use out_of_scope. Never invent order IDs, amounts, dates, or policies "
    "not present in the input."
)
user = ("Complaint: I was charged twice for my headphones on the same day.\n"
        "Transaction record: order_id: AE-44021 | amount: AED 300.00 | "
        "note: two authorizations of AED 75.00 on 2026-05-11")

messages = [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False, pad_token_id=tok.eos_token_id)
raw = tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()
print(json.dumps(json.loads(re.search(r"\{.*\}", raw, re.S).group(0)), indent=2))

Limitations

Decision-support only (recommends, never executes actions); trained on illustrative synthetic policy data; small evaluation set; 4-bit QLoRA training implies minor numerical differences from full precision. See sections 12–14 for details.

License

Apache 2.0, consistent with the Qwen3-Base license.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.