illate

qwen3-4b-banking77-lora

README

License: apache-2.0

Results (full test set, 3,080 messages)

Table with columns: Model, Accuracy, 95% CI, Macro-F1, Invalid outputs
Model	Accuracy	95% CI	Macro-F1	Invalid outputs
TF-IDF + logistic regression (plain baseline, trains on a CPU)	89.3%	88.1% to 90.3%	89.3%	0
This adapter	94.0%	93.1% to 94.8%	94.0%	0
Qwen3-4B, same model, no fine-tuning (served, label list in prompt)	63.1%

Difference against the baseline: +4.7 points (paired bootstrap 95% CI +3.7 to +5.8).
Served with vLLM on one NVIDIA L4, labels constrained: 93.9% accuracy. One request at a time: p50 194 ms, p95 269 ms. 32 in flight: about 87 requests/s, p95 0.55 s. Latency was measured in the same region as the GPU; add your own network round trip.
Confidence intervals are bootstrap resamples over the test messages.

Most common mistakes are near-duplicates in the label set, for example fiat_currency_support read as exchange_via_app (5 times) and card_arrival as card_delivery_estimate (4 times).

Replacing a fine-tuned gpt-4.1-nano

Added 10 October 2026. Measured by ILLATE, 5 to 8 October 2026 (UTC); the runs with the 20 most similar examples were pre-registered on all three test sets.

OpenAI is scheduled to stop serving fine-tunes of gpt-4.1-nano on 23 October 2026 and names gpt-5.6-luna as the replacement base model (deprecations page, checked 8 October 2026). Luna's model page lists fine-tuning as "Not supported" (GPT-5.6 Luna, checked 8 October 2026), so the like-for-like replacement is a prompt. If you are replacing a nano classifier, these are the options we measured.

Banking77: the same 3,080 test messages as this adapter

Table with columns: Model, Prompt or setup, Accuracy, 95% CI, This adapter minus it (paired 95% CI), Macro-F1
Model	Prompt or setup	Accuracy	95% CI	This adapter minus it (paired 95% CI)	Macro-F1
This adapter (Qwen3-4B + LoRA)	fine-tuned, one L4	94.0%	93.1 to 94.8		94.0%
GPT-5.6 Luna	20 most similar training examples, JSON schema, reasoning none	93.8%	92.9 to 94.6	+0.2 (-0.6 to +0.9), tied	93.8%
bge-base embeddings + logistic regression

Three options are statistically tied on this task: GPT-5.6 Luna with the 20 most similar training examples in its prompt, bge-base (and bge-small) plus logistic regression, and this adapter. Their paired intervals include zero. On a task like this, accuracy doesn't pick between them; cost at your volume, latency and control do. The embeddings classifier needs no GPU to serve. Luna with retrieval is pay per call, about 1,600 input tokens a call here, and needs a labelled pool to search.
The examples did the work for Luna. One fixed example per category gave GPT-5.6 Luna 87.2%; the 20 most similar gave 93.8%. Reasoning effort low didn't help, and GPT-6 Luna scored below GPT-5.6 Luna in every Banking77 setting; on CLINC150 with the examples the two were level (94.4% against 94.6%, paired +0.2, -0.2 to +0.6, post-hoc).
The 20-example runs were pre-registered: settings, metric and the publish rule were committed before any full run. Luna with the JSON schema and no examples didn't fit that run's budget and was not run.
Setup:
- The GPT rows other than the Decisions API row used OpenAI's Chat Completions API. The 20-example runs held answers to the label list with Structured Outputs; the examples were picked from the training split by bge-small-en-v1.5 cosine similarity. The earlier runs had no schema; answers were matched to the label list: both Lunas at reasoning effort none and the default temperature, GPT-4.1-mini at temperature 0.
- The Decisions API ran one choice question per message; its 10 refusals count as wrong.
- gpt-oss-120b ran with vLLM at low reasoning effort, with outputs constrained to the label list.
- Accuracy CIs come from 10,000 bootstrap resamples. The differences use a paired bootstrap on the same test items. One run per setting.

CLINC150 and CFPB

The same comparison on two more test sets, with the same pre-registered 20-example Luna setup (no adapters for these are published here):

Table with columns: Test set, Qwen3-4B + LoRA, Best embeddings classifier, GPT-5.6 Luna, 20 most similar examples, GPT-6 Luna
Test set	Qwen3-4B + LoRA	Best embeddings classifier	GPT-5.6 Luna, 20 most similar examples	GPT-6 Luna
CLINC150: 5,500 requests, 150 intents + out-of-scope	95.1%	90.6% (bge-base)	94.6%	94.4% (20 most similar examples)
CFPB complaints: 5,117, 10 products (renamed products merged, a merge chosen after seeing the errors)	89.5%	86.4% (bge-small)	87.5%	86.0% (category list only; not run with examples, budget)

CLINC150 is a tie: the fine-tuned model minus GPT-5.6 Luna with examples is +0.5 points (paired 95% CI -0.04 to +1.0). What's left is out-of-scope handling, and it's a trade-off: the fine-tuned model caught more out-of-scope requests (83.9% against 79.1%, paired +4.8, +2.5 to +7.2) and wrongly rejected slightly more real ones (0.4% against 0.2%, 18 of 4,500 against 8). GPT-6 Luna with the category list only caught 90.0% but wrongly rejected 5.5%.
CFPB, by product: on credit reporting (3,266 of the 5,117 complaints) the fine-tuned model got 117 more right, more than the whole gap of 104. Counted as complaints right, it was also ahead on debt collection (21) and checking and savings (3), and GPT-5.6 Luna on mortgages (12), credit cards (11), vehicle loans (7), payday and personal loans (6) and money transfer (1); the other two products were level. By F1 (post-hoc, no intervals) Luna was ahead on payday and personal loans, and the two were level on credit cards, checking and savings, and mortgages. Overall the fine-tuned model leads by +2.0 points (paired 95% CI +1.2 to +2.9), which clears the 2-point line set before the run by about two complaints. On the 14 products as filed: 67.0% for the fine-tuned model, 62.3% for Luna with examples, and 67.7% for TF-IDF, which ties it.
The embeddings classifiers trailed the fine-tuned model on both sets: by at least 4.5 points on CLINC150, mostly on out-of-scope requests, and by 3.1 to 3.9 points on CFPB.

Not measured

a fine-tuned gpt-4.1-nano itself (OpenAI refused our fine-tuning job on 7 October 2026);
reasoning effort medium, or Structured Outputs without examples;
the Decisions API with a description for each choice;
a fine-tuned encoder such as ModernBERT.

Full method, latency and limits: illate.dev/luna-vs-fine-tuned-model. Raw numbers: bakeoff.json. Not sure whether your own fine-tune is on OpenAI's list: illate.dev/fine-tune-shutdown-checker.

Training

Data: Banking77 training set (CC BY 4.0, PolyAI), 9,000 messages for training and 1,003 held out as a dev set; the official 3,080-message test set is untouched.
LoRA rank 32, alpha 32, dropout 0.05, on all attention and MLP projections.
2 epochs, learning rate 2e-4 with cosine schedule, batch 16, bf16, one NVIDIA L4 (about 22 minutes).
Labels are human-written dataset labels; no outputs from closed models were used.

How to use

The model answers with the label name. Use the same system prompt as in training:

python

from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base = "Qwen/Qwen3-4B-Instruct-2507"tok = AutoTokenizer.from_pretrained(base)model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")model = PeftModel.from_pretrained(model, "illate/qwen3-4b-banking77-lora") messages = [    {"role": "system", "content": "Classify the bank customer's message. Reply with the category name only."},    {"role": "user", "content": "I still haven't received my new card, when will it arrive?"},]ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)out = model.generate(ids, max_new_tokens=16, do_sample=False)print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))  # card_arrival

The reported numbers use constrained decoding, which only lets the model write one of the 77 label names. Without it, the model can occasionally produce a label that does not exist; with vLLM, pass the label list as a structured-output choice.

With vLLM:

bash

vllm serve Qwen/Qwen3-4B-Instruct-2507 --enable-lora --lora-modules banking77=illate/qwen3-4b-banking77-lora

Limits

Trained for Banking77's 77 intents only; it is not a general banking assistant.
English only.
Benchmark data is public and short (one message each). Accuracy on your own traffic will differ; measure it before relying on it.

License

Adapter: Apache 2.0, same as the base model. Training data: Banking77, CC BY 4.0 (Casanueva et al., 2020, "Efficient Intent Detection with Dual Sentence Encoders").

Contact: poojith@illate.dev · illate.dev

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

illate

Model Tree

Base

Qwen/Qwen3-4B-Instruct-2507

Adapter

this model

Input Modalities

Text

Output Modalities