🔑 TL;DR — what changed vs. the base model
We measured the model on six standard finance benchmarks before and after training.
Table with columns: Capability, Benchmark, Base, Finsight, Change| Capability | Benchmark | Base | Finsight | Change |
|---|
| 🟢 Math over financial tables | FinQA | 0.019 | 0.126 | ▲ 6.6× |
| 🟢 Multi-turn financial math | ConvFinQA | 0.148 | 0.203 | ▲ +37% |
| 🟡 News sentiment | FPB | 0.812 | 0.794 | ▼ −2% |
| 🟡 Headline classification | Headlines | 0.745 | 0.721 | ▼ −3% |
| 🟡 Entity extraction | NER (F1) | 0.238 | 0.180 | ▼ −24% |
| 🔴 Aspect sentiment | FiQA-SA | 0.681 | 0.370 | ▼ −46% |
| Average | 0.441 | 0.399 | ▼ −0.042 |
In plain English:
- ✅ Big win on numerical reasoning. The model got dramatically better at
answering questions that require doing math over the tables in financial filings
(FinQA went from ~2% to ~13% accuracy — a 6.6× jump).
- ⚠️ Classification got worse, mostly one task (FiQA-SA aspect sentiment).
- 📊 No net average gain (−0.042). This is shared as an honest, reproducible
experiment, not a polished product — the numbers tell a real story (see
Why the mixed results below).
🛠️ What we did
A clean, controlled experiment — same benchmark suite run before and after, only the
training changes:
Qwen3-14B-Base
│
▼ Stage 1: Continued Pre-training (CPT) ~330M tokens of raw finance text
Finsight-CPT
│
▼ Stage 2: Supervised Fine-tuning (SFT) 191k finance instruction examples
Finsight-CPT-SFT ◄── this model
│
▼ Re-run the exact same benchmarks and compare
Decision from the numbers
Both stages used QLoRA (memory-efficient 4-bit training with LoRA adapters),
then the adapters were merged back into full 16-bit weights.
📚 Datasets we used
Stage 1 — Continued pre-training (raw text, "learn to read finance")
~330M tokens, decontaminated against every benchmark test set so scores stay honest:
Table with columns: Source, What it is| Source | What it is |
|---|
SEC 10-K filings (anonymous-md/EDGAR_FILINGS_DATASET) | Full annual-report text — the core of financial disclosure |
Financial news (ashraq/financial-news-articles) | Market/company news articles |
Industry finance corpus (BAAI/IndustryCorpus_Finance) | Long-form finance/economics text |
Stage 2 — Instruction tuning (Q&A pairs, "learn to answer finance")
191,428 instruction examples, also decontaminated:
Table with columns: Source, What it is| Source | What it is |
|---|
Josephgflowers/Finance-Instruct-500k | Broad finance instruction following |
sujet-ai/Sujet-Finance-Instruct-177k | Mixed finance tasks / QA |
gbharti/finance-alpaca | General finance instruction pairs |
FinGPT/fingpt-sentiment-train, FinGPT/fingpt-fiqa_qa | Sentiment + FiQA QA (train splits only) |
Decontamination: any training example overlapping a benchmark test item
(exact match or ≥50% 13-gram overlap) was removed from both datasets, so the
benchmark scores above are not inflated by memorization.
📏 Benchmarks we used
Evaluated with lm-evaluation-harness
on the FinBen flare_* finance suite, using the public
ChanceFocus/flare-* datasets. All three
variants (Base / CPT / final) were scored on the exact same samples with greedy
decoding — a fair, apples-to-apples comparison.
Table with columns: Benchmark, Skill tested, Metric| Benchmark | Skill tested | Metric |
|---|
| FinQA | Numerical reasoning over financial tables | accuracy |
| ConvFinQA | Multi-turn numerical reasoning | accuracy |
| FPB (Financial PhraseBank) | News sentiment (3-class) | accuracy |
| Headlines | Gold-news headline classification | accuracy |
| FiQA-SA | Aspect-based sentiment | accuracy |
|
🧭 Why the mixed results
- CPT helped reasoning but shifted the model's "voice." Teaching it on raw
filings/news sharpened math-over-tables, but nudged its calibration on simple
classification.
- The FiQA-SA drop is partly a measurement artifact. The final model learned a
chat format, but the multiple-choice benchmarks are scored on raw text likelihood
without the chat template — which disadvantages chat-tuned models. Re-scoring
with the chat template applied is the top recommended next step and would likely
recover much of the classification gap.
- This was a lightweight CPT (a LoRA adapter over ~0.2B tokens). A heavier /
full-parameter continued pre-training would likely shift the trade-off.
🚀 How to use
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "JahangirAtMaxint/Finsight-Qwen3-14B-CPT-SFT"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
messages = [{"role": "user", "content": "What does a rising debt-to-equity ratio tell an investor?"}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=256)
print(tok.decode(out[0], skip_special_tokens=True))
The model uses a ChatML chat template — use apply_chat_template for
instruction-style prompts.
⚙️ Training details
- Base: Qwen/Qwen3-14B-Base (Apache-2.0, pre-training-only)
- CPT: QLoRA (4-bit NF4 + LoRA r=64, α=128), packed seq len 2048, effective batch
128, lr 2e-4 cosine, 800 steps (~0.2B tokens), Flash-Attention 2.
- SFT: QLoRA (r=32, α=64) on the CPT weights, ChatML template, packed seq len 2048,
effective batch 128, 2 epochs.
- Hardware: 8× NVIDIA A100 SXM4 40 GB (AWS p4d.24xlarge). End-to-end ~20 h
(CPT ~5 h, SFT ~2 h, benchmarking the rest).
- Reproducibility: the full run report, minute-by-minute timeline, and raw
per-benchmark eval JSONs are in the
finsight_run/ folder of this repo.
⚠️ Limitations
Research artifact, not production-ready. No net benchmark lift on average; not
safety-aligned. Not financial advice — always verify numerical outputs.