Results
Evaluated with lm-evaluation-harness (HF backend) on the FinBen flare_* finance
tasks over the public ChanceFocus/flare-* datasets. Identical sample counts across
all three variants (fair comparison), greedy decoding.
Table with columns: Task (metric), Base, CPT, SFT (this model), Δ (SFT−Base)| Task (metric) | Base | CPT | SFT (this model) | Δ (SFT−Base) |
|---|
| flare_finqa (acc) — numerical reasoning | 0.0192 | 0.0820 | 0.1264 | +0.1072 |
| flare_convfinqa (acc) — multi-turn numeric | 0.1483 | 0.1463 | 0.2034 | +0.0550 |
| flare_fpb (acc) — sentiment | 0.8124 | 0.7897 | 0.7938 | −0.0186 |
| flare_headlines (acc) — classification | 0.7453 | 0.7137 | 0.7206 | −0.0247 |
| flare_ner (f1) — entity extraction | 0.2377 | 0.1500 | 0.1796 | −0.0582 |
| flare_fiqasa (acc) — aspect sentiment | 0.6809 | 0.3191 | 0.3702 | −0.3106 |
| Average | 0.4406 | 0.3668 | 0.3990 | −0.0416 |
Takeaways
- Numerical/tabular reasoning improved markedly — FinQA nearly 7× (0.019 → 0.126),
ConvFinQA +37%. The finance corpus + instruction data genuinely helped math over
filings/tables.
- Classification regressed, dominated by the FiQA-SA collapse (0.68 → 0.37).
- CPT alone hurt every task (avg 0.44 → 0.37); SFT recovered part of it (→ 0.40),
but not back to baseline on classification.
Training pipeline
Table with columns: Stage, What, Data, Method| Stage | What | Data | Method |
|---|
| 1. Baseline | Benchmark the untouched base | — | lm-eval HF backend |
| 2. CPT | "No-instruction" domain adaptation | ~330M-token raw finance corpus (SEC 10-K filings + financial news + industry text), decontaminated vs. the flare_* test sets | QLoRA (4-bit NF4 + LoRA r=64, α=128), next-token prediction on packed raw text, 800 steps (~0.2B tokens) |
| 3. SFT | Instruction tuning on the CPT weights | 191,428 finance instruction rows (Finance-Instruct-500k, Sujet-Finance, finance-alpaca, FinGPT sentiment/FiQA) | QLoRA (r=32, α=64), ChatML template, 2 epochs |
Both LoRA stages were merged to 16-bit; this repo is the final merged CPT→SFT model.
Key hyperparameters
- CPT: 4-bit NF4 base, LoRA r=64/α=128/dropout=0.05 on all linear projections;
seq len 2048 (packed), effective batch 128 sequences, lr 2e-4 cosine, adamw_8bit,
gradient checkpointing, Flash-Attention 2.
- SFT: 4-bit NF4 CPT base, LoRA r=32/α=64; seq len 2048 packed, effective batch 128,
lr 2e-4 cosine, 2 epochs, ChatML chat template.
- Decontamination: training rows overlapping any
flare_* test item (exact or
≥50% 13-gram shingle overlap) were dropped from both corpora.
The run — ~20 hours on 8× A100
Executed on a single AWS p4d.24xlarge (8× NVIDIA A100 SXM4 40 GB, NVSwitch),
unattended end-to-end.
Table with columns: Phase, Duration, Notes| Phase | Duration | Notes |
|---|
| Data prep (CPT corpus + SFT mixture) | ~15 min | CPU-only, streamed + decontaminated |
| Baseline eval | ~1.5 h | 8-GPU data-parallel |
| CPT (QLoRA, 800 steps) | ~5 h | 8× A100 @ 99%, ~22 GB/card, 22 s/step |
| SFT (QLoRA, 314 steps) + merge | ~1 h 52 m | on the CPT weights |
| Re-benchmark (base-known + CPT + SFT) | ~8.7 h | generation tasks dominate |
|
Engineering notes (things that broke and were fixed)
Getting a very new stack (transformers 5.x / TRL 1.x / accelerate 1.x, CUDA-13
driver, no system CUDA toolkit) working on 8× A100 required several fixes:
- Full-parameter CPT was abandoned. DeepSpeed ZeRO-3 needs
nvcc (no toolkit on
the small root disk), and PyTorch FSDP would not shard the model in this
TRL/transformers/accelerate combination — every rank held the full model and OOM'd
identically across 7 configurations. CPT was therefore done as QLoRA, which is
what this model reflects.
- QLoRA + data-parallel device placement: bitsandbytes placed every rank's 4-bit
model on GPU 0 → OOM. Fixed with a per-process
device_map.
- Eval backend: vLLM 0.24 hard-requires FlashInfer (JIT needs a CUDA toolkit), so
evaluation uses lm-eval's HF (transformers) backend, data-parallel across GPUs.
- Finance eval tasks shipped as in-repo lm-eval YAML over the public
ChanceFocus/flare-* datasets (the PIXIU/FinBen fork is incompatible with the stack).
- Generation cap: the base
generation_config defaults to max_new_tokens=2048;
capped to 256 for the short-answer generation tasks (applied equally to all variants).
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "JahangirAtMaxint/Qwen3-14B-Finance-CPT-SFT"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
messages = [{"role": "user", "content": "Summarize the key risks in a 10-K risk-factors section."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=256)
print(tok.decode(out[0], skip_special_tokens=True))
The SFT stage installed a ChatML chat template; use apply_chat_template for
instruction-style prompts.
Caveats & limitations
- No net benchmark lift (avg −0.042). Improvements are concentrated in numerical
reasoning; classification (esp. FiQA-SA) regressed.
- Likely eval-format artifact: the multiple-choice classification tasks are scored
by raw log-likelihood without applying the chat template, which disadvantages a
chat-tuned model. Part of the classification regression may be measurement, not
capability — re-evaluating with the chat template applied is a recommended next step.
- QLoRA-CPT is light: r=64 over ~0.2B tokens nudged the base's calibration without
adding enough durable domain knowledge to pay for it; full-parameter CPT (not
runnable here) may behave differently.
- Not safety-tuned or aligned for production; not financial advice. Verify all
numerical outputs.
Citation / provenance
Domain-adapted from Qwen3-14B-Base with the Finsight pipeline
(baseline → CPT → SFT → re-benchmark). Trained 2026-07-04/05 on 8× A100.