JahangirAtMaxint

Qwen3-14B-Finance-CPT

README

License: apache-2.0

Base: Qwen/Qwen3-14B-Base (Apache-2.0)
Method: QLoRA (4-bit NF4 base + LoRA r=64, α=128, dropout 0.05 on all linear projections), next-token prediction on packed raw text.
Corpus: ~330M-token finance mix — SEC 10-K filings + financial news + industry text — decontaminated against the flare_* benchmark test sets.
Schedule: seq len 2048 (packed), effective batch 128 sequences, lr 2e-4 cosine, adamw_8bit, 800 steps (~0.2B tokens), Flash-Attention 2, gradient checkpointing.
Hardware: 8× A100 SXM4 40 GB (AWS p4d.24xlarge), ~5 h.

Table with columns: Task, Base, CPT, SFT (final)
Task	Base	CPT	SFT (final)
flare_finqa	0.0192	0.0820	0.1264
flare_convfinqa	0.1483	0.1463	0.2034
flare_fpb	0.8124	0.7897	0.7938
flare_headlines	0.7453	0.7137	0.7206
flare_ner	0.2377	0.1500	0.1796
flare_fiqasa	0.6809	0.3191	0.3702
Average	0.4406	0.3668	0.3990

See the final model card for the full pipeline, engineering notes, and caveats.

Not financial advice. Verify all outputs.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Container

Run this model inference with full control and performance in your environment.

Model Details

Model Provider

JahangirAtMaxint

Model Tree

Base

Qwen/Qwen3-14B-Base

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

License: apache-2.0

Base: Qwen/Qwen3-14B-Base (Apache-2.0)
Method: QLoRA (4-bit NF4 base + LoRA r=64, α=128, dropout 0.05 on all linear projections), next-token prediction on packed raw text.
Corpus: ~330M-token finance mix — SEC 10-K filings + financial news + industry text — decontaminated against the flare_* benchmark test sets.
Schedule: seq len 2048 (packed), effective batch 128 sequences, lr 2e-4 cosine, adamw_8bit, 800 steps (~0.2B tokens), Flash-Attention 2, gradient checkpointing.
Hardware: 8× A100 SXM4 40 GB (AWS p4d.24xlarge), ~5 h.

Table with columns: Task, Base, CPT, SFT (final)
Task	Base	CPT	SFT (final)
flare_finqa	0.0192	0.0820	0.1264
flare_convfinqa	0.1483	0.1463	0.2034
flare_fpb	0.8124	0.7897	0.7938
flare_headlines	0.7453	0.7137	0.7206
flare_ner	0.2377	0.1500	0.1796
flare_fiqasa	0.6809	0.3191	0.3702
Average	0.4406	0.3668	0.3990

See the final model card for the full pipeline, engineering notes, and caveats.

Not financial advice. Verify all outputs.