GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
johnsonfzzz
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
johnsonfzzz
Model Tree
Qwen/Qwen2.5-1.5B-Instruct
Input Modalities
Output Modalities
Supported Functionality
The test complaints were filed after every training complaint (time-based split), the way a deployed model meets new data. Each product has 300 test complaints, so no single product dominates.
Same 1,000 test complaints for every model:
| Model | Product | Product + issue | Avg input tokens | p50 latency | Cost / 1k requests |
|---|---|---|---|---|---|
| This model | 77.5% | 50.0% | 295 | 0.99s | $0.0072 |
| Claude Sonnet 5.5 | 75.6% | 48.5% | 2,533 | 1.39s | $2.82 |
| Claude Haiku 5.5 | 75.2% | 46.1% | 2,533 | 2.34s | $0.46 |
| TF-IDF + logistic regression, same 20,000 examples | 69.9% | 47.1% | ~$0 | ||
| Qwen2.5-1.5B-Instruct, not fine-tuned | 47.4% | 5.5% | 1,626 | 0.96s | $0.0067 |
On all 3,300 test complaints, this model scores 77.8% on product and 49.1% on product and issue, and every answer is valid JSON with a real label.
high. Their costs are as reported by the API.Why product + issue is only about 50% for every model: customers choose the product and issue themselves when they file, so the labels are noisy, and many issues overlap (the label set even contains both Problem with a company's investigation into an existing problem and ...existing issue). Product accuracy is the more reliable number.
The model's probability for its whole answer works as a confidence score. Thresholds were chosen on 1,093 validation complaints and measured on the 3,300 test complaints:
| Routing by product: target accuracy | Handled automatically | Accuracy on those |
|---|---|---|
| 85% | 77.6% | 84.8% |
| 90% | 57.8% | 88.8% |
| 95% | 24.8% | 94.7% |
| Product + issue: target accuracy | Handled automatically | Accuracy on those |
|---|---|---|
| 70% | 47.6% | 69.9% |
| 80% | 28.9% | 80.3% |
| 90% | 11.5% | 92.1% |
Use this exact system prompt. The model was trained on it, and the labels aren't listed in it. A different prompt will lower accuracy.
text
Classify this consumer financial complaint. Reply with JSON: {"product": ..., "issue": ...}Use greedy decoding (do_sample=False or temperature=0) and max_new_tokens=80.
python
import jsonfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "johnsonfzzz/Qwen2.5-1.5B-CFPB-Complaint-Routing"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto") messages = [ {"role": "system", "content": 'Classify this consumer financial complaint. Reply with JSON: {"product": ..., "issue": ...}'}, {"role": "user", "content": "I received a gift card for $100.00 from a friend. Its an American Express gift card. It wont spend in-person or online."},]inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)output = model.generate(**inputs, max_new_tokens=80, do_sample=False)answer = json.loads(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))print(answer)# {'product': 'Prepaid card', 'issue': 'Trouble using the card'}python
from peft import PeftModelfrom transformers import AutoModelForCausalLM base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct", dtype="auto")model = PeftModel.from_pretrained(base, "johnsonfzzz/Qwen2.5-1.5B-CFPB-Complaint-Routing", subfolder="lora")bash
vllm serve johnsonfzzz/Qwen2.5-1.5B-CFPB-Complaint-Routing --served-model-name cfpb --max-model-len 4096python
import jsonimport mathfrom openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")resp = client.chat.completions.create( model="cfpb", messages=[ {"role": "system", "content": 'Classify this consumer financial complaint. Reply with JSON: {"product": ..., "issue": ...}'}, {"role": "user", "content": "A debt collector keeps calling me about a loan I never took out."}, ], temperature=0, max_tokens=80, logprobs=True,)answer = json.loads(resp.choices[0].message.content)confidence = math.exp(sum(t.logprob for t in resp.choices[0].logprobs.content))print(answer, round(confidence, 3)) # send to a person if confidence is below your thresholdThe 11 products are: Checking or savings account; Credit card; Credit reporting or other personal consumer reports; Debt collection; Debt or credit management; Money transfer, virtual currency, or money service; Mortgage; Payday loan, title loan, personal loan, or advance loan; Prepaid card; Student loan; Vehicle loan or lease. Issues are the CFPB's own issue names for each product, 132 product–issue pairs in all.
| Setting | Value |
|---|---|
| Base model | Qwen/Qwen2.5-1.5B-Instruct |
| Method | Supervised fine-tuning with LoRA (TRL + PEFT), loss on the JSON answer only |
| LoRA | r=16, alpha=32, dropout=0.05, all linear layers |
| Optimizer | AdamW, learning rate 2e-4, cosine schedule, 20 warm-up steps |
| Batch size / epochs | 16 / 2 (2,500 steps); the checkpoint with the lowest validation loss (step 2,250) was kept |
| Precision | bf16 |
| Max length | 1,024 tokens; complaints were cut to 500 words (8% were longer) |
| Hardware | 1× NVIDIA L40S on Modal, 82 minutes |
Data. Complaints with a written narrative from July to December 2025, deduplicated first: 49% of narratives were exact or near-exact copies (often the same template letter filed many times), which would otherwise leak between training and test.
| Split | Filed | Size |
|---|---|---|
| Train | Jul 1 – Oct 15, 2025 | 20,000, balanced across products |
| Validation | Oct 16 – Oct 31, 2025 | 1,093 |
| Test | Nov 1 – Dec 31, 2025 | 3,300 (300 per product) |
The model is released under Apache 2.0, the license of the base model. Training data comes from the CFPB Consumer Complaint Database, published by the US government. Consumers consent to publishing their narratives, and the CFPB removes personal information before release (shown as XXXX).
import jsonfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "johnsonfzzz/Qwen2.5-1.5B-CFPB-Complaint-Routing"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto") messages = [ {"role": "system", "content": 'Classify this consumer financial complaint. Reply with JSON: {"product": ..., "issue": ...}'}, {"role": "user", "content": "I received a gift card for $100.00 from a friend. Its an American Express gift card. It wont spend in-person or online."},]inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)output = model.generate(**inputs, max_new_tokens=80, do_sample=False)answer = json.loads(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))print(answer)# {'product': 'Prepaid card', 'issue': 'Trouble using the card'}from peft import PeftModelfrom transformers import AutoModelForCausalLM base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct", dtype="auto")model = PeftModel.from_pretrained(base, "johnsonfzzz/Qwen2.5-1.5B-CFPB-Complaint-Routing", subfolder="lora")vllm serve johnsonfzzz/Qwen2.5-1.5B-CFPB-Complaint-Routing --served-model-name cfpb --max-model-len 4096import jsonimport mathfrom openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")resp = client.chat.completions.create( model="cfpb", messages=[ {"role": "system", "content": 'Classify this consumer financial complaint. Reply with JSON: {"product": ..., "issue": ...}'}, {"role": "user", "content": "A debt collector keeps calling me about a loan I never took out."}, ], temperature=0, max_tokens=80, logprobs=True,)answer = json.loads(resp.choices[0].message.content)confidence = math.exp(sum(t.logprob for t in resp.choices[0].logprobs.content))print(answer, round(confidence, 3)) # send to a person if confidence is below your threshold