The Problem

Financial institutions process thousands of customer complaints daily across mobile apps, websites, contact centres, email, and regulatory portals. These complaints arrive as free-form text and must be manually categorized before investigation and resolution can begin.

The result is predictable:

• Complaints are routed to the wrong operational teams
• Manual review effort increases
• Resolution times become longer
• Customer experience deteriorates
• Regulatory complaint handling becomes more expensive

Traditional classifiers typically predict one label at a time and struggle with the nuanced language used in consumer finance complaints.

This project addresses the problem as a structured generation task. A single model call extracts all required complaint taxonomy fields simultaneously.

---

What the Model Does

Given a customer complaint narrative, the model generates:

json
Copy code
{
  "product": "Checking or savings account",
  "sub_product": "Checking account",
  "issue": "Unauthorized transactions or other transaction problem",
  "sub_issue": "Debit card issue"
}

These fields map directly to the CFPB complaint taxonomy and can be consumed by routing systems, workflow engines, complaint management platforms, and analytics pipelines.

---

Model Details

---

Training Configuration

LoRA Adapter

Only LoRA adapter weights were updated during training.

Training Hyperparameters

---

Training Convergence

Validation loss steadily decreased from 1.709 → 1.598, demonstrating successful adaptation of the base model to the CFPB complaint taxonomy.

---

Dataset

Source: CFPB Consumer Complaint Database

The model was trained to predict four operational complaint fields:

1. Product
2. Sub-Product
3. Issue
4. Sub-Issue

The task is formulated as structured JSON generation rather than independent classification.

---

Inference with Constrained Decoding

Inference uses a two-stage approach:

Stage 1

The fine-tuned model generates structured JSON.

Stage 2

Generated values are aligned to the nearest canonical CFPB label using TF-IDF similarity matching.

This improves robustness when the model generates labels that are semantically correct but differ slightly from official CFPB terminology.

---

Evaluation Results

Evaluated on 250 held-out CFPB complaints.

Baseline refers to the original Qwen2.5-7B-Instruct model without fine-tuning.

Product Classification Performance

Sub-Product Semantic Similarity

Issue Semantic Similarity

Sub-Issue Semantic Similarity

---

Final Results Summary

Run inference

python
Copy code
def categorise_complaint(complaint_text: str, model, tokenizer) -> dict:
    messages = [
        {
            "role": "system",
            "content": (
                "You are a banking complaint classification assistant. "
                "Given a consumer complaint narrative, extract the CFPB ticket fields "
                "as a JSON object with keys: product, sub_product, issue, sub_issue."
            ),
        },
        {
            "role": "user",
            "content": complaint_text,
        },
    ]

    prompt = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=128,
            do_sample=False,
            pad_token_id=tokenizer.eos_token_id,
        )

    prompt_len = inputs["input_ids"].shape[1]
    generated  = tokenizer.decode(output[0][prompt_len:], skip_special_tokens=True)
    return generated


complaint = """
I reported fraudulent transactions on my debit card and the bank reversed
my provisional credit without explaining the investigation outcome.
"""

result = categorise_complaint(complaint, model, tokenizer)
print(result)
# {"product": "Checking or savings account", "sub_product": "Checking account",
#  "issue": "Unauthorized transactions or other transaction problem",
#  "sub_issue": "Debit card issue"}

---

Dependencies

markdown
Copy code
transformers==4.44.0
peft==0.12.0
accelerate==0.34.0
datasets==2.21.0
torch (ROCm-compatible build for AMD, or standard CUDA build)
scikit-learn
rouge-score
sacrebleu
nltk

---

Limitations

• CFPB taxonomy only. The model is trained on and constrained to CFPB Consumer Complaint Database labels. It is not a general-purpose complaint classifier and should not be used with complaint taxonomies from other regulatory bodies or internal systems without retraining.
• Issue field accuracy. The issue field (33.6% accuracy) is the weakest link. The CFPB issue taxonomy contains 80+ canonical strings with overlapping phrasing. Expanding training data and further tuning the constrained decoder are the most direct paths to improvement.
• English language only. All training data is in English. Performance on non-English complaints is untested and likely poor.
• Context length. Complaints longer than 1024 tokens will be truncated. Most CFPB complaints are well within this limit, but very long narratives may lose relevant context.

---

Intended Use

This model is intended for use by:

• Banking operations teams automating first-touch complaint categorisation
• Compliance teams processing regulatory complaint filings
• Contact centre platforms routing incoming complaints before agent assignment
• Research teams studying LLM adaptation for financial NLP tasks

It is not intended for consumer-facing deployment without human review of outputs, or for use in jurisdictions where automated complaint classification decisions have legal or regulatory implications without appropriate oversight.

---

Training Infrastructure

Trained on an AMD Instinct MI300X GPU (192 GB HBM3 VRAM) running ROCm 7.2.4. The training stack is fully ROCm-native — bitsandbytes (CUDA-only) is not used. Model precision is bfloat16, which is the native compute type for the CDNA3 architecture.

---

Citation

If you use this model in research or production, please cite the CFPB Consumer Complaint Database as the data source:

markdown
Copy code
Consumer Financial Protection Bureau (CFPB)
Consumer Complaint Database
https://www.consumerfinance.gov/data-research/consumer-complaints/