Results
Banking77 is scored on the full official test set (3,080 examples). The base model gets the same
prompt (labels in the prompt) and the same greedy decoding as the fine-tuned model. MMLU and GSM8K are
checks for side effects, on random subsets of their test sets.
Table with columns: Suite, n, Base, With this adapter, Change (points), 95% CI of change (paired bootstrap)| Suite | n | Base | With this adapter | Change (points) | 95% CI of change (paired bootstrap) |
|---|
| Banking77 test | 3,080 | 69.58% (2,143) | 82.82% (2,551) | +13.25 | [+11.88, +14.64] |
| Banking77 dev (held out from training) | 308 | 67.21% (207) | 82.14% (253) | +14.94 | [+10.39, +19.48] |
| MMLU, 500 random test questions | 500 | 76.40% | 77.20% | +0.80 | [0.00, +1.80] |
| GSM8K, 400 random test problems | 400 | 95.25% | 95.75% | +0.50 | [0.00, +1.25] |
- Banking77 test, exact McNemar p = 7.8e-82. Items the base got right and the adapter got wrong: 54.
Items the base got wrong and the adapter got right: 462.
- A second interval that resamples the 77 classes instead of the examples: [+8.99, +17.73] points.
- Macro-F1 on the test set: 0.690 to 0.826. Share of outputs that are a valid label: 99.03% to 99.97%.
- Scoring the answers strictly or leniently gave the same accuracy for both models.
- MMLU and GSM8K: the intervals include zero, so the data show no measurable change on these subsets. They
do not prove that nothing else changed.
- One training run, one seed. Seed-to-seed variation was not measured.
All numbers in the table are copied from eval/SCORE.json, written by a scoring script
that was run once on hashed inputs. The base model in these measurements is the pre-quantised checkpoint
unsloth/Qwen3-32B-bnb-4bit (revision
7f721e74a6a8cc9ee352f7e49303a2c1705f9083, NF4) loaded with Transformers and bitsandbytes (its own
quantisation config, bfloat16) with this adapter applied through PEFT. Nothing was measured on a BF16 base.
Control: the same recipe without layer streaming
As a cross-check, the same recipe (rank 8, same data order, same optimiser and schedule, 300 steps) was
trained with ordinary Hugging Face + PEFT on the resident published 4-bit checkpoint (micro-batch 2). It is informational and
decided nothing. These numbers come from a separate comparison output that is not part of this repository.
Table with columns: Layer streaming (this adapter), Resident HF + PEFT control | Layer streaming (this adapter) | Resident HF + PEFT control |
|---|
| Banking77 test accuracy | 82.82% (2,551 / 3,080) | 83.18% (2,562 / 3,080) |
| Change vs base (95% CI) | +13.25 [+11.88, +14.64] | +13.60 [+12.24, +14.97] |
| Training process, peak allocated / reserved GPU memory | 5,052 / 5,216 MiB | 20,127 / 24,236 MiB |
The control is 0.36 points higher on the test set (paired bootstrap CI [+0.03, +0.68], exact McNemar
p = 0.052; the two models produced identical generated token ids on 3,033 of 3,080 items). The difference
was not isolated; the LoRA initialisation, micro-batching and kernel paths differ between the two setups.
The memory comparison is a rough order of magnitude (about 4x): the micro-batch and the memory cap were
not the same in the two runs.
How it was trained
- Data: 1,200 distinct rows from the Banking77 train split (15 to 16 per class). Rows whose normalised
text also occurs in the test set were removed before sampling, so none of the 1,200 training rows is a
test row. A 308-row dev split (4 per class) was held out of training.
- Recipe: one pass over the 1,200 rows = 300 optimizer steps of 4 rows, no gradient accumulation,
maximum length 576 tokens. LoRA r = 8, alpha = 16, dropout 0 on
q_proj and v_proj of all 64 layers
(256 adapter tensors, 9,961,472 parameters). AdamW, learning rate 1e-4, weight decay 0,
gradient clipping at 1.0, 10 warmup steps and a cosine decay to 1e-5. LoRA initialisation seed 32017.
The loss is computed on the label tokens plus <|im_end|> only. Mean loss per 25-step window went from
0.81 (steps 1 to 25) to 0.19 (steps 276 to 300).
- Base during training: Soup quantised the BF16
Qwen/Qwen3-32B weights itself to NF4
(blocksize 64, double quantisation) and streamed them layer by layer from pinned host RAM.
- LoRA precision: the adapter was kept in fp32 by a small launcher shim. Without it, the trainer stack
(TRL with 4-bit loading) leaves LoRA parameters in bfloat16. The saved adapter is fp32 (39.9 MB).
- Launcher shims: Soup's own model, loss, optimiser and checkpointing were not modified. A small launcher
script around
soup train did four things: capped the PyTorch allocator, applied the exact learning-rate
schedule above (the stock cosine schedule would decay to 0 rather than 1e-5), forced sequential
sampling of the rows in file order, and logged every step. The logged learning rate matched the
formula at all 300 steps.
- Where it ran, and what that does and does not show:
- Training ran on a rented NVIDIA L40S (48 GB) with the PyTorch allocator capped at 6.25 GiB to
simulate an 8 GB card budget. Peak allocated memory was 5,052 MiB, peak reserved 5,216 MiB, and the
whole-device peak sampled during training was about 5.8 GiB (5,805 MiB by
nvidia-smi).
- The cap only limits the PyTorch allocator, so the CUDA context and library workspaces sit outside it.
That is why the whole-device number is reported next to it.
- It did not run on an 8 GB card. That training fits an 8 GB budget is an inference from the cap
and the sampled device peak. Step times (median 2.65 s, 300 steps in about 13.4 minutes) belong to the
L40S and do not transfer to other hardware.
- The only run on a consumer 8 GB GPU (RTX 5070 Laptop) was a forward-only reload of this adapter
through Soup's streaming engine: 32 probe items, 2,492 MiB peak allocated, mean cross-entropy 0.1395
against 0.1401 on the training card. No training step has been run on an 8 GB card.
- Software: torch 2.14.0+cu130, transformers 5.17.0, peft 0.20.0, trl 0.29.1, bitsandbytes 0.50.2,
accelerate 1.15.0.
The adapter expects the prompt it was trained with: a single user turn that lists the 77 labels
(in Python sorted() order, one per line) followed by the customer message, rendered with the Qwen3 chat
template with thinking disabled (an empty <think> block). The model then answers with the label only.
The label names come from the dataset; note that Refund_not_showing_up is capitalised there and sorts first.
import torchfrom datasets import load_datasetfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer BASE = "unsloth/Qwen3-32B-bnb-4bit" # the 4-bit checkpoint the results above were measured onADAPTER = "AlpamysSigmaaaa/Qwen3-32B-Banking77-LoRA-Soup" labels = sorted(load_dataset("PolyAI/banking77")["train"].features["label"].names) # 77 names PROMPT = ( "Classify the customer message into exactly one of the 77 intent labels below.\n" "Answer with the label only, exactly as it is written in the list.\n" "\n" "Labels:\n" "{labels}\n" "\n" "Customer message: {text}") tokenizer = AutoTokenizer.from_pretrained(BASE)model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map={"": 0})model = PeftModel.from_pretrained(model, ADAPTER)model.eval() def classify(text: str) -> str: user_message = PROMPT.replace("{labels}", "\n".join(labels)).replace("{text}", text) chat = tokenizer.apply_chat_template( [{"role": "user", "content": user_message}], add_generation_prompt=True, enable_thinking=False, # the empty <think></think> block the adapter was trained with tokenize=False, ) inputs = tokenizer(chat, return_tensors="pt", add_special_tokens=False).to(model.device) with torch.no_grad(): output = model.generate(**inputs, max_new_tokens=24, do_sample=False) answer = tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True) return answer.strip() print(classify("I still have not received my new card, I ordered over a week ago."))
Notes:
- The prompt-building code above was checked token for token against the prompt builder used for training
and evaluation. The loading and generation lines mirror the evaluation harness but were not re-run in
exactly this form. The 4-bit base weights are about 19 GB, so plan for a GPU with well over 20 GB; the
evaluation processes sampled up to about 25,300 MiB of device memory on a 48 GB card.
transformers 5.x takes dtype=; older versions use torch_dtype=.
- Evaluation used greedy decoding and at most 24 new tokens for Banking77. If the answer is not exactly one
of the 77 labels (this happened for 0.03% of test outputs of the fine-tuned model), treat it as invalid.
- MMLU was scored zero-shot by comparing the logits of the letters A to D; GSM8K by greedy decoding of up to
512 tokens with an answer line of the form
#### <number>.
- Training and evaluation used the base prompted in non-thinking mode only. Behaviour with thinking enabled
was not tested.
Limitations and how to read the numbers
- One seed, one run. The size of seed-to-seed variation is unknown. The confidence intervals above
describe the sampling of examples for this one adapter, not repeated training.
- One task, one language, one adapter. English Banking77 only, rank 8 on
q_proj and v_proj. Nothing
is claimed about other tasks, other quantisations, BF16 serving, full fine-tuning or other model sizes.
- Training base and evaluation base differ slightly. The adapter was trained against Soup's own NF4
quantisation of the BF16 weights and evaluated on
unsloth/Qwen3-32B-bnb-4bit. In the three layers that
were compared, the packed 4-bit weights are byte-identical and the per-block scale statistics differ
at the 1e-7 relative level. The effect on outputs was checked on 32 probe items (mean absolute
cross-entropy difference between the streaming engine and Hugging Face: 0.0047 for the fine-tuned model,
0.0282 for the base).
- Banking77 may have been seen by Qwen3 in pretraining. If so, the base score may be inflated. This was
not checked.
- A protocol existed before the runs. It fixed the data, prompts, metrics, thresholds and pass/fail
rules in advance (SHA-256
a00f1c1f29b2657ffc7c311b75bc09e86514d2c3c78872906ee600b032c8f120). The
scorer applied those rules once to the final outputs. The result passes the gates that were evaluated:
Banking77 gain of at least 5 points with a positive interval (+13.25, lower bound +11.88), the lower
bound of the interval of change at least -3 points on MMLU (0.00) and at least -5 points on GSM8K (0.00),
and byte-identical greedy outputs when the evaluation was repeated in a new process.
- The run deviated from the protocol in 23 documented places (D1 to D23). The log is not part of this
repository. The ones that matter when reading the numbers:
- training ran on a rented 48 GB card with a 6.25 GiB allocator cap instead of on an 8 GB GPU, and used
Soup's
main branch with a launcher shim instead of the engine named in the protocol;
- the training base was quantised by Soup, the evaluation base was the published checkpoint (above);
- the protocol's in-process reload check (G3a) was replaced by a weaker one: two fresh processes loading
the saved adapter into the streaming engine produced identical logits hashes and cross-entropy on all
32 probe items;
- the freeze receipt of the code and data hashes was written after the base model had been evaluated on
the 308-example dev split, and before any test-set or fine-tuned evaluation;
- one base-model test evaluation ran while a 3-step launcher check shared the GPU; the greedy outputs of a
later base-model repeat on the dev split were byte-identical, but the test run was not repeated;
- with the cached-prefix evaluation path, the fine-tuned model matched the full-prompt path on 63 of 64
check items.
The protocol's strict reading treats a changed recipe or an unevaluated check as "no verdict". The
deviations were reviewed and accepted by the people who ran the experiment, on the condition that every
description of the result states where training ran, which engine was used, the LoRA precision, the
freeze timing, the control and the single seed; this card does so.
- No claim that it is the best 77-class classifier. The run used 1,200 of the 10,003 training examples
and was not compared with other models trained on Banking77.
Provenance
Table with columns: Item, Value| Item | Value |
|---|
| Adapter file | adapter_model.safetensors, fp32, 39,880,048 bytes, SHA-256 1a9847ec4ed14a3a2b9fa3be1ba5c6ab9700898fdccd6475214b86dbb4c12659 |
| Soup commit | 74e6628fc27454a6d82502b2d8b2e68b6bf159da |
| Training base | Qwen/Qwen3-32B, revision 9216db5781bf21249d130ec9da846c4624c16137 (BF16), quantised to NF4 by Soup |
| Evaluation base | unsloth/Qwen3-32B-bnb-4bit, revision 7f721e74a6a8cc9ee352f7e49303a2c1705f9083 |
| Banking77 | PolyAI/banking77, revision 1846984189f2bc0ac51c6fc3a600fa9e1714e0c6 |
| MMLU, GSM8K (evaluation only) | cais/mmlu revision c30699e8356da336a370243923dbaf21066bb9fe, openai/gsm8k revision 740312add88f781978c0658806c59bc2815b9866 |
| Scorer output | eval/SCORE.json |
adapter_config.json points base_model_name_or_path at Qwen/Qwen3-32B. The training run used a local
copy of that checkpoint, so this field was rewritten from the local directory name to the Hub id before
upload; the weights file is unchanged.
License
The adapter is released under Apache-2.0, the same license as the base model
Qwen/Qwen3-32B. The training data,
PolyAI/banking77, is licensed under CC-BY-4.0
(Casanueva et al., 2020, "Efficient Intent Detection with Dual Sentence Encoders",
arXiv:2003.04807); attribution is given here. MMLU and GSM8K were used
for evaluation only and are licensed under MIT.