Evaluation — base vs adapter (honest, reproducible)
Held-out 10% of the aq22 instruction set (120 samples, seed 42), ROUGE
scoring, max 96 new tokens.
Table with columns: Metric, Base, Adapter, Δ| Metric | Base | Adapter | Δ |
|---|
| rouge1 | 0.2485 | 0.2539 | +0.0054 |
| rouge2 | 0.0807 | 0.0683 | -0.0124 |
| rougeL | 0.2022 | 0.2198 | +0.0176 |
Verdict: small but real gains on ROUGE-1 (+0.005) and ROUGE-L (+0.018);
a slight ROUGE-2 regression (−0.012) — consistent with the small 135M base
and CPU training budget. The adapter makes outputs measurably more
concise/on-topic for financial instructions, but this is a learning
artifact, not a production model.
Side-by-side generations (held-out)
1. You are a financial analyst categorizing tweets into specific financial topics. Given a tw...
- Reference: Stock Commentary
- Base: The topic name for this tweet is "TIL Cell Therapy Set To Prove Itself."
- Tuned: Stock Market
2. You are a financial sentiment analysis expert. Your task is to analyze the sentiment expre...
- Reference: positive
- Base: Positive
- Tuned: Positive
3. You are a financial analyst identifying sentiment towards specific entities in financial n...
- Reference: The sentiment in this text about McDonald is positive. The sentiment in this tex
- Base: McDonald is a positive sentiment.
McDonald is a positive sentiment because it indicates that the company is i
- Tuned: McDonald is a positive sentiment.
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")
model = PeftModel.from_pretrained(base, "vivekkopthsd/finlora-adapter")
Limitations
- 135M base — bounded reasoning capacity
- Mixed eval deltas (ROUGE-2 regression) — verify per use case
- Single-domain financial instruction data; not safety-tuned for general chat