Gydeai

complaint-classifier-in-rbi

README

License: apache-2.0

What it outputs

json

{"category": "deposit_accounts", "entity_type": "bank", "tags": ["kyc_records", "account_freeze"]}

category, one of the 10 the RBI complaint portal (cms.rbi.org.in) lists for a bank: Opening/Operation of Deposit accounts, ATM/CDM/Debit card, Credit Card, Mobile / Electronic Banking, Loans and Advances, Notes and Coins, Remittance and Collection of instruments, Pension related, Para-Banking, Other products and services. For an NBFC the portal lists 7: Notes and Coins, Remittance and Collection of instruments and Pension related are bank-only.
entity_type: bank, nbfc or unclear.
tags: what went wrong, for example charges_fees, recovery_conduct, unauthorised_transaction, kyc_records. RBI has no separate category for charges, recovery agents, KYC or mis-selling; they sit under the product, and the tags carry them.

categories.json has the ids, names, definitions and tags. Category names are RBI's; definitions, tags and rules are Gyde's.

Evaluation

463 synthetic complaints written by Claude Sonnet 5.5 across all categories, banks and NBFCs, many personas and styles (formal, angry, Hinglish, on behalf of a parent). A complaint counts only if three AI labellers agreed on its category, with each other and with the category it was written for (455 unanimous, 8 two-of-three with that category). These are silver labels: no person reviewed them.

Table with columns: Model, Category accuracy, Category macro-F1, Bank / NBFC, Tags F1
Model	Category accuracy	Category macro-F1	Bank / NBFC	Tags F1
Claude Sonnet 5.5, prompt only (full instructions)	99.4%	0.992	98.0%	0.869
GPT-5.4, prompt only (full instructions, default)	99.1%	0.989	96.0%	0.847
This adapter (Qwen3-8B + LoRA, rank 16)	96.3%	0.966	99.6%	0.787
Qwen3-8B, 5 retrieved training complaints as examples	95.0%	0.944	97.8%	0.764
Qwen3-8B, prompt only (full instructions)	91.4%	0.917	96.2%	0.684

Read these numbers with three caveats:

Frontier models from two providers score about 99% on these categories. Sonnet wrote the complaints and Claude Opus helped label them, but GPT-5.4, which had no part in building the set, also scores 99.1%. The RBI's 10 categories are well defined, and general-purpose models already know them: on category, this adapter is genuinely about 3 points behind. Its case here is cost, data residency and the bank/NBFC field (99.6%). Where labels follow an organisation's own conventions, training matters: on the 82 US complaint issues, the US adapter is 16 to 21 points ahead of GPT-5.4 (us-cfpb).
Agreement removes the hardest cases. 37 complaints on which the labellers disagreed are held out; on those the adapter picks one of the labellers' choices 34 times (agreement, not accuracy).
Synthetic hard cases were not very hard. A further 143 complaints were written as hard cases and labelled the same way (without one of the three labellers, so it could be compared fairly elsewhere). Written to a stated category, they mostly stay clear: Sonnet scored 143/143, GPT-5.4 and this adapter 142/143.

Hard-case set (143), category accuracy by case:

Table with columns: Case, Complaints, This adapter, Qwen3-8B, 5 examples, Claude Sonnet 5.5, prompt only, GPT-5.4, prompt only
Case	Complaints	This adapter	Qwen3-8B, 5 examples	Claude Sonnet 5.5, prompt only	GPT-5.4, prompt only
Hindi (Devanagari)	20	20	20	20	20
Noisy (typos, SMS style, Hinglish)	19	19	17	19	19
One line	19	19	19	19	19
Rambling, problem buried	15	14	13	15	15
Two issues, one main	22	22	21	22	21
Bank or NBFC not stated	16	16	16	16	16
Near-miss categories (e.g. account problem met in the app)	23	23	22	23	23
Firm outside the Ombudsman scheme	9	9	8	9	9
All	143	142 (99.3%)	136 (95.1%)	143 (100%)	142 (99.3%)

The adapter's remaining confusions on the 463: Opening/Operation of Deposit accounts read as Mobile / Electronic Banking (4), Loans and Advances read as Deposit accounts (4), Pension related read as Deposit accounts (2).

Cost per 1,000 complaints (USD)

Table with columns: Model, Real-time, Batch
Model	Real-time	Batch
This adapter, self-hosted	about the US adapter's ~$0.012	~ $0.014 t o$ 0.034 (as US)
Claude Sonnet 5.5, prompt only	~$6.0	~$1.0
GPT-5.4, prompt only	~$4.5	~$2.2

Estimates. Sonnet 5.5 at list prices ( $2/$ 10 per million input/output tokens, cached reads $0.20) o n t h e t o k e n s m e a s u r e df or t h ese 463 co m pl ain t s (ab o u t 2, 550 in p u t, o f w hi c hab o u t 2, 240 a r e t h es ha r e d in s t r u c t i o n s, an d 94 o u tp u tp er co m pl ain t) : r e a l - t im e w i t h o u t c a c hin g; ba t c ha tt h e 50 p r i ces ($ 2.50/ $15 p er mi l l i o nin p u t / o u tp u tt o k e n s) o n t h e t o k e n s m e a s u r e d h er e (ab o u t 1, 570 in p u t, 35 o u tp u t; n oc a c h e d in p u tw a sr e p or t e d), ba t c ha t i t s 50 p r o m pt (ab o u t 260 t o k e n s) i ss h or t er t han t h e U S a d a pt e r' s, so t h e U S f i g u r es (o n e N V I D I AA 30 in I n d iaa t ab o u t$ 0.41/hour, about 10 complaints/s when busy) are a fair guide. At low volume a dedicated GPU costs about $9.80/day whatever the traffic.

Use

Base model Qwen/Qwen3-8B (revision b968826d9c46dd6066d109eabc6255188de91218). Greedy decoding (temperature 0), max_tokens 160, thinking disabled. The adapter was trained on exactly this prompt:

System: You file Indian bank and NBFC customer complaints under the RBI Integrated Ombudsman Scheme. Reply with JSON only: {"category": ..., "entity_type": ..., "tags": [...]}.
User: Complaint: {text}\nAnswer:

bash

vllm serve Qwen/Qwen3-8B --revision b968826d9c46dd6066d109eabc6255188de91218 \  --enable-lora --max-lora-rank 16 --lora-modules in-rbi=Gydeai/complaint-classifier-in-rbi

python

from openai import OpenAIclient = OpenAI(base_url="http://localhost:8000/v1", api_key="local")SYSTEM = ('You file Indian bank and NBFC customer complaints under the RBI Integrated Ombudsman Scheme. '          'Reply with JSON only: {"category": ..., "entity_type": ..., "tags": [...]}.')r = client.chat.completions.create(    model="in-rbi", temperature=0, max_tokens=160,    extra_body={"chat_template_kwargs": {"enable_thinking": False}},    messages=[{"role": "system", "content": SYSTEM},              {"role": "user", "content": "Complaint: UPI payment of Rs 4,500 failed but money was debited from my "                                          "savings account 10 days ago, still not reversed.\nAnswer:"}])print(r.choices[0].message.content)   # {"category": "mobile_electronic_banking", "entity_type": "bank", ...}

Validate the answer against categories.json (a category id, and for an NBFC not one of the three bank-only ones); constrained JSON decoding makes an invalid answer impossible.

Training

6,404 synthetic complaints written by Reflection Beam (Beam-501B-A23B) from the category definitions, with personas, styles and channels; Beam then classified each one without being told its intended category, and only matching ones were kept (7,115 of 7,775; 711 held out for validation). No evaluation text was used to write them, and no output of Claude or of the other labellers was used for training. Labels are the category and entity type each complaint was written for, and the tags Beam reported. LoRA rank 16, alpha 32, dropout 0.05 on all attention and MLP projections; 1 epoch, learning rate 2e-4; about 25 minutes on one A100 40 GB (run.json). Training size: 2,964 complaints gave 93.7%, 4,444 gave 95.2%, 6,404 gave 96.3% on the 463.

Limits

No real-complaint evaluation yet. Real complaints are messier than synthetic ones; expect lower accuracy.
Indian bank and NBFC complaints only. Entities not covered by the Ombudsman scheme (for example peer-to-peer lending platforms) go to RBI's Consumer Education and Protection Cell; the adapter has no output for that and will still pick a category.
English, Hinglish and some Hindi in training; mainly English in evaluation.
For routing and triage, not for deciding a customer's case or any credit decision. A person reviews outcomes.
Base model origin: Qwen (Alibaba Cloud). A version on another base model is available on request.

Data and licence

All complaints are synthetic; no real customer data was used. Adapter: Apache 2.0. Base model: Qwen3-8B under its own licence (Apache 2.0). Category names follow the RBI complaint portal; this adapter is not affiliated with or endorsed by the Reserve Bank of India.

Gyde serves this model inside India with data-residency, policy and failover controls, and trains versions on your own categories and complaints: gyde.ai.

Built by Gyde (DAI Labs Private Limited, Pune, India).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Gydeai

Model Tree

Base

Qwen/Qwen3-8B

Adapter

this model

Input Modalities

Text

Output Modalities