Model lineage
Table with columns: Stage, Model, Notes| Stage | Model | Notes |
|---|
| Base | Qwen/Qwen2.5-0.5B | Pre-trained checkpoint |
| This checkpoint | andreadm/reddit-pulse-qwen2.5_0.5b-qdora | QDoRA+/xQDoRA+ PEFT adapter on Reddit titles, three-way head down / neutral / up |
Labels
Table with columns: id, label, encoding used in the paper's corpus files| id | label | encoding used in the paper's corpus files |
|---|
| 0 | down | -1 |
| 1 | neutral | +0 |
| 2 | up | +1 |
How to use
The adapter is loaded on top of the 4-bit quantized base model, exactly as
it was trained (bitsandbytes and peft required). config.json in this
repository carries the three-way head, the label names and the pad token,
so no argument beyond the repository name is needed:
import torch
from peft import PeftModel
from transformers import (
AutoConfig,
AutoModelForSequenceClassification,
AutoTokenizer,
BitsAndBytesConfig,
)
name = "andreadm/reddit-pulse-qwen2.5_0.5b-qdora"
tokenizer = AutoTokenizer.from_pretrained(name)
tokenizer.padding_side = "left"
config = AutoConfig.from_pretrained(name)
base = AutoModelForSequenceClassification.from_pretrained(
"Qwen/Qwen2.5-0.5B",
config=config,
dtype=torch.bfloat16,
quantization_config=BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
),
device_map="auto",
)
model = PeftModel.from_pretrained(base, name).eval()
encoding = {"down": -1, "neutral": 0, "up": 1}
texts = [
"Inflation expectations drop to lowest level since 2021, NY Fed survey shows",
"What is the difference between CPI and PCE?",
]
with torch.inference_mode():
batch = tokenizer(texts, padding=True, truncation=True, max_length=1024, return_tensors="pt")
ids = model(**batch.to(model.device)).logits.argmax(dim=-1).tolist()
print([encoding[config.id2label[i]] for i in ids])
Do not hand the repository name to AutoModelForSequenceClassification
directly: transformers' adapter shortcut rebuilds this DoRA adapter with
different logits than the trained model.
Intended use and limitations
Intended use. Labelling large volumes of short, informal, English,
economy-related texts (Reddit titles and comments, headlines, social media
posts) with a directional inflation signal that is then aggregated over
time — the paper's use case. The model is a building block for a
high-frequency indicator, not a stand-alone oracle.
Limitations.
- Single predictions are noisy. The value of the model comes from
averaging thousands of predictions per period, where idiosyncratic errors
wash out. Do not rely on any single label.
- Domain and register. Trained on r/economy, r/Economics and
r/wallstreetbets titles about US inflation from 2008 to 2022. Other
countries, other registers and post-2022 vocabulary are out of distribution.
- Class imbalance.
down is the minority class of the gold set and the
hardest one; the loss was class-balanced during training, but down
recall remains lower.
- Short texts. Fine-tuned on titles (median 11 words, max 52). Long
comments are truncated and were not seen during training.
- Direction, not stance or sentiment. The model does not say whether the
author wants inflation to move, nor whether the news is good or bad;
only which way prices are said to be going.
Training data
The gold set is a hand-labelled sample of 1,383 Reddit submission titles
from r/economy, r/Economics and r/wallstreetbets, dated February 2008 to
December 2022. Labels were produced by a human-in-the-loop protocol (manual
annotation assisted by zero-shot LLaMA-70B, a fine-tuned LLaMA-8B classifier
and ChatGPT-assisted adjudication of disagreements) described in the paper.
The gold set itself is not redistributed here.
Table with columns: label, titles, share| label | titles | share |
|---|
| neutral | 623 | 45.0 % |
| up | 537 | 38.8 % |
| down | 223 | 16.1 % |
Training procedure
Seed protocol and model selection
The paper's protocol asks how sensitive the classifier is to which titles
it is trained on, so it separates two sources of randomness:
- 19 split seeds each draw a different stratified
71 / 19 / 10 % train / validation / test
partition (982 / 263 / 138 titles) of the same gold set.
- One fixed seed governs everything else: adapter and head
initialisation, batch shuffling and dropout. Every split therefore trains
the same model the same way on different data.
Each split is fine-tuned independently; the checkpoint with the median
test weighted-F1 across the runs (the upper median) is kept and the others
discarded. The result is a typical run, not the best one. This checkpoint
is split seed 107935903.
Hyperparameters
Table | |
|---|
| Base model | 4-bit NF4, double-quantized (bitsandbytes), frozen |
| Adapters | DoRA (use_dora), rank r = 32, alpha = 32, dropout = 0.1, on k_proj, o_proj, q_proj, v_proj; classification head trained in full |
| Optimizer | AdamW (fused) with LoRA+ (adapter B matrices at 5x the base learning rate) |
| Learning rate | 0.0001, cosine decay, no warm-up |
| Objective |
The exact TrainingArguments are in training_args.json
and the governing configuration extract in
training_config.yml.
Evaluation
Selected checkpoint (split seed 107935903)
Table with columns: split, accuracy, F1 weighted, F1 macro, precision macro, recall macro, ROC-AUC| split | accuracy | F1 weighted | F1 macro | precision macro | recall macro | ROC-AUC |
|---|
| validation | 0.745 | 0.734 | 0.681 | 0.723 | 0.668 | 0.843 |
| test | 0.669 | 0.661 | 0.587 | 0.595 | 0.585 |
Split sensitivity across the 19 seeds
Test metrics of every run, sorted by weighted F1; the selected checkpoint is marked.
Table with columns: seed, accuracy, F1 weighted, F1 macro, precision macro, recall macro, ROC-AUC| seed | accuracy | F1 weighted | F1 macro | precision macro | recall macro | ROC-AUC | |
|---|
| 477284336 | 0.755 | 0.743 | 0.689 | 0.748 | 0.673 | 0.852 | |
| 2786505123 | 0.734 | 0.730 | 0.703 |
The full tables are in evaluation/.
Files
Table with columns: file, content| file | content |
|---|
adapter_config.json | PEFT adapter configuration (base model, rank, target modules) |
adapter_model.safetensors | DoRA adapter weights and the three-way classification head (base weights are not redistributed) |
chat_template.jinja | chat template inherited from the base tokenizer (unused by the classifier) |
config.json | architecture, three-way head and label map |
tokenizer.json | tokenizer |
|
Reproducing
git clone https://github.com/andrea-dm/reddit-pulse && cd reddit-pulse
uv venv && uv pip install -e .
reddit run --model qwen2.5_0.5b --gpu 0 # every split seed, median selection, corpus labelling
reddit upload --model qwen2.5_0.5b # this repository, from the selected checkpoint
The gold set (data/labelled.xlsx) and the subreddit corpus are not part of
the repository; see the paper for the data-construction stages.
Citation
If you use this model, please cite the paper it was built for:
@article{delmonaco2026reddit,
title = {Reddit's `pulse' on {US} inflation: forecasting with large language models},
author = {Del Monaco, Andrea and Longo, Luigi and Marcucci, Juri and Tafani, Irene},
journal = {Journal of Applied Econometrics},
year = {2026},
note = {forthcoming},
}
@techreport{delmonaco2026reddit_qef,
title = {Reddit's `pulse' on {US} inflation: forecasting with large language models},
author = {Del Monaco, Andrea and Longo, Luigi and Marcucci, Juri and Tafani, Irene},
institution = {Banca d'Italia},
series = {Questioni di Economia e Finanza (Occasional Papers)},
number = {1028},
year = {2026},
month = jun,
doi = {10.32057/0.QEF.2026.1028},
}
License
Apache License, Version 2.0, inherited from the base model. The upstream notices are included: LICENSE.
The views expressed in the paper are those of the authors and do not
necessarily reflect those of the Bank of Italy, the Eurosystem, or the
European Commission.