What this stage did
We continued pre-training the base model on ~330M tokens of raw finance text so it
would better understand financial language before any instruction tuning:
Table with columns: Source, What it is| Source | What it is |
|---|
SEC 10-K filings (anonymous-md/EDGAR_FILINGS_DATASET) | Full annual-report text |
Financial news (ashraq/financial-news-articles) | Market/company news |
Industry finance corpus (BAAI/IndustryCorpus_Finance) | Long-form finance text |
Trained with QLoRA (4-bit NF4 + LoRA r=64, α=128), 800 steps (~0.2B tokens), then
merged to 16-bit. Corpus decontaminated against the benchmark test sets.
Results (on the FinBen flare_* finance benchmarks)
Same samples, greedy decoding, via lm-evaluation-harness.
Table with columns: Benchmark (skill), Base, CPT (this), Final (CPT→SFT)| Benchmark (skill) | Base | CPT (this) | Final (CPT→SFT) |
|---|
| FinQA — table math | 0.019 | 0.082 | 0.126 |
| ConvFinQA — multi-turn math | 0.148 | 0.146 | 0.203 |
| FPB — sentiment | 0.812 | 0.790 | 0.794 |
| Headlines — classification | 0.745 | |
Reading this: CPT alone improved FinQA numerical reasoning (0.019 → 0.082) but
lowered the average — continued pre-training on raw text sharpened reasoning while
disrupting classification calibration. The following SFT stage recovers part of the
gap. See the final model card
for the full pipeline, datasets, benchmarks, and analysis.
Training details
- Base: Qwen/Qwen3-14B-Base (Apache-2.0)
- Method: QLoRA CPT (r=64, α=128), packed seq len 2048, effective batch 128,
lr 2e-4 cosine, 800 steps, Flash-Attention 2.
- Hardware: 8× A100 SXM4 40 GB (AWS p4d.24xlarge), ~5 h.
Not financial advice. Verify all outputs.