Training Details
Training Data
Dataset: FinGPT/fingpt-forecaster-dow30-202305-202405
Covers all 30 DOW Jones Index constituents over May 2023 – May 2024. Each sample is a weekly snapshot including:
- Company profile — name, sector, IPO date, market cap, exchange, ticker
- News — headlines and summaries from the past 1–4 weeks (via Finnhub)
- Basic financials — ~35 ratios (EPS, P/E, gross margin, ROE, debt ratios, etc.)
- Price history — weekly open/close prices for the lookback window
- Answer — structured output with positive developments, potential concerns, a % price movement prediction, and analysis (GPT-4 generated labels)
Data is sourced from yfinance and Finnhub. Split: 80% train / 20% test.
Training Procedure
Supervised fine-tuning (SFT) of LLaMA-3-8B-Instruct using LoRA. Dataset prompts are originally in LLaMA-2 chat format ([INST]<<SYS>>...[/INST]) and are automatically converted to the LLaMA-3 chat template via tokenizer.apply_chat_template before tokenization. Prompt tokens are masked from the loss; only answer tokens are supervised.
Preprocessing
- Tokenizer:
meta-llama/Meta-Llama-3-8B-Instruct; pad_token = eos_token, padding_side = right
- Prompt format: LLaMA-3 chat template (system + user messages), with
add_generation_prompt=True
- Max sequence length: 8192 tokens
- Samples exceeding max length are filtered out before training
- LoRA parameters explicitly cast to fp32 for gradient scaler compatibility
Training Hyperparameters
Table with columns: Hyperparameter, Value| Hyperparameter | Value |
|---|
| Training regime | fp16 mixed precision |
| Optimizer | AdamW |
| Learning rate | 5e-5 |
| LR scheduler | Constant (with warmup) |
| Warmup ratio | 0.03 |
| Weight decay | 0.01 |
| Batch size per device | 1 |
| Gradient accumulation steps | 4 |
| Number of epochs | 5 |
LoRA Configuration:
Table with columns: Parameter, Value| Parameter | Value |
|---|
| Rank (r) | 8 |
| Alpha | 16 |
| Dropout | 0.1 |
| Bias | none |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Task type | CAUSAL_LM |
Speeds, Sizes, Times
- Hardware: Multi-GPU SLURM cluster (4 GPUs per node), Apptainer container
- Checkpoint cadence: Every 10% of training steps
Evaluation
Metrics
Evaluation is run at each checkpoint via GenerationEvalCallback on 50 randomly sampled test examples (greedy decoding, max 512 new tokens, stopping at <|eot_id|> or EOS).
Table with columns: Metric, Description| Metric | Description |
|---|
| Binary Accuracy | Directional accuracy — did the model correctly predict up vs. down? |
| MSE | Mean squared error on predicted % price movement |
| ROUGE-1/2/L | N-gram overlap per section (positive developments, potential concerns, analysis) |
| BERTScore | Semantic similarity (precision, recall, F1) computed per section |
Parse rate (% of outputs matching the expected structured format) is also logged.
Results
Table with columns: Model, Dir. Acc., MSE, ROUGE| Model | Dir. Acc. | MSE | ROUGE |
|---|
| FinGPT (Llama-3) | 0.6122 | 7.2653 | 0.2467 |
| FinGPT (Llama-2) | 0.5102 | 9.7142 | 0.2425 |
| Llama-3 (base) | 0.4568 | 19.9748 | 0.2387 |
| Llama-2 (base) | 0.4201 | 28.4471 | 0.2023 |
|
Technical Specifications
Model Architecture and Objective
- Base: LLaMA-3-8B-Instruct (decoder-only causal LM, 8B parameters)
- Adaptation: LoRA on all attention projection layers and MLP gate/up/down projections
- Objective: Next-token prediction (cross-entropy) over the structured answer only; prompt is masked
- Key difference vs LLaMA-2 version: Uses the native LLaMA-3 chat template and
<|eot_id|> as an additional stop token; gradient checkpointing uses use_reentrant=False
Compute Infrastructure
Multi-GPU SLURM cluster with Apptainer containers. Training coordinated via torchrun with NCCL backend and DeepSpeed ZeRO-2. Weights & Biases for experiment tracking.
Hardware
- NVIDIA A100 GPUs (4 per node)
Software
Table with columns: Package, Notes| Package | Notes |
|---|
| PEFT 0.11.0 | LoRA implementation |
| HuggingFace Transformers | Model and trainer |
| PyTorch | Distributed via torchrun |
| DeepSpeed | ZeRO-2 optimization |
| HuggingFace Datasets | Data loading |
| WandB | Experiment tracking |
Citation
BibTeX:
@misc{fingpt2023,
title={FinGPT: Open-Source Financial Large Language Models},
author={Yang, Hongyang and Liu, Xiao-Yang and Wang, Christina Dan},
journal={FinLLM Symposium at IJCAI 2023},
year={2023}
}