Jainamshahhh

flashfacts-4b

README

License: apache-2.0

Read this first: what kind of number follows

Adapters are usually scored by win rate, the share of rows where a language model judge preferred the tuned output. Every number on this card is exact-match accuracy against figures taken from the filing itself.

This entry is the one that can prove the difference matters, because it ran both. The identical corpus was trained on Adaption's own AutoScientist, and that platform reported a 95.48% win rate. We downloaded that adapter and scored it with our scorer on our slice: 62.0% exact-match accuracy. Same corpus, same base, same adapter. A win rate is a preference; an accuracy is a fact, and here they differ by 33 points.

Table

what is measured	exact match against the figures in the filing
a language model judged something	never, not the labels, not the scoring
synthetic training rows	zero, all 13,976 are real SEC filings
held-out rows scored	650, across three independent slices
rows the base model wins	0 of 650
significance	p = 1.1e-109, exact McNemar, held-out
shortcut gain from column position	0.00, measured, not asserted
traceability	every row carries its SEC accession number and a SHA256 of the source

TL;DR

On held-out filings the base model scores 0.0% and FlashFacts-4B scores 90.8%. On an out-of-distribution anchor drawn from 10-Q filings, a different document type entirely, it scores 74.0%.

The number worth reading twice is not the headline. It is that this corpus has no positional shortcut, so the score cannot be earned by guessing.

The shortcut test

The obvious way to fake financial extraction is positional: always take the first numeric column. The build measures that directly, on the released corpus.

Table

accuracy from column position alone	57.58%
majority-class baseline	57.58%
shortcut gain	0.00

A model that sees only which column a number sits in does exactly as well as always guessing the most common answer, and no better. An earlier build leaked 0.189 of shortcut gain and it was engineered out before training. Any score on this task was earned by reading the table.

Headline results

Base and tuned generated in one process under identical greedy decoding, scored by the released score_flashfacts.py, which a reviewer can run on a laptop with no GPU.

Table with columns: slice, rows, what it isolates, base, FlashFacts-4B
slice	rows	what it isolates	base	FlashFacts-4B
`ff_held`	400	held-out filings, companies and periods disjoint from training	0.0%	90.8%
`ff_hard`	150	scale and period disambiguation together	0.0%	91.3%
`ff_ood`

Significance, stated rather than assumed

Base and tuned answer the same rows in one process, so this is a paired design and the right test is an exact McNemar test on the discordant rows, those where exactly one model was right. The null is that the adapter is no better than the base.

Table with columns: slice, only tuned right, only base right, exact two-sided p
slice	only tuned right	exact two-sided p
`ff_held`	363	1.1e-109
`ff_hard`	137	1.1e-41
`ff_ood`	74	1.1e-22

The base wins no row on any slice, including the out-of-distribution 10-Q anchor. Recompute with the released significance.py from the released generation files.

Abstention is graded, not assumed

Every row in the training corpus has at least one genuinely absent field. In this domain a system that invents a plausible revenue figure when the filing does not state one is worse than useless, so returning null correctly is scored as its own outcome rather than folded into accuracy.

Table with columns: base, FlashFacts-4B
	base	FlashFacts-4B
correctly returned `null` for an absent field	96.5%	99.9%
invented a value for an absent field	2.1%	0.1%
emitted parseable JSON	98.5%	100.0%

A second arm, trained on Adaption's own platform

The identical corpus was also trained through Adaption's AutoScientist, on their compute, and the resulting adapter was downloaded and evaluated on our slice with our scorer under identical greedy decoding, so the two arms are directly comparable.

Table with columns: arm, training, ff_held
arm	training	`ff_held`
ours	one A100, 3 epochs, lr 1e-4, max_len 2048	90.8%
platform	AutoScientist, 5 epochs, lr 1e-5, batch approximately 550	62.0%

The platform reported its own win rate of 95.48% on iteration 1 of 3. That figure is not an accuracy: a win rate is the share of rows where the tuned output beats the base, and this base scores near zero on its own distribution, so beating it is not the same as being right. Both numbers are published, and only the column measured with our scorer is comparable to anything else on this card.

Why this base

google/gemma-3-4b-it is the base the AutoScientist platform itself trains, which is what makes the platform arm above directly comparable: same base, same corpus, two training paths. It is also the base of every entry in this portfolio, fixed before any result existed.

Its 0.0% headline deserves one sentence of honesty in each direction. The base is not globally broken, and this card reports where it is competent: 96.5% correct nulls and 98.5% parseable JSON. What it cannot do is land all four fields of the strict contract at once, and that is the task. The claim here is the absolute score against that contract, not the ratio to the base, which is why the abstention and format tables report the base's real abilities beside the tuned model's.

Scope and operating notes

Scope. US registrants filing in English with the SEC, 2018 to 2024, EX-99 earnings exhibits. Revenue and period extraction specifically. Not guidance, not sentiment, not forward-looking statements, and not investment advice.
Token budget is real. The median row is 1,716 tokens and the maximum is 2,000. A 1024 window truncates 100% of rows and 1536 truncates 80.2%. This model requires a 2048 context; run it shorter and it will fail on the input, not on the task.
Decontamination has a declared gap. Checked clean against FinQA, ConvFinQA and TatQA across 3,250 reference questions, maximum 4-gram Jaccard 0.0151. FinanceBench was named in the build spec but was not available locally, so it is not covered. Stated rather than omitted.
920 rows were dropped at build time for exceeding the token budget, rather than trained on truncated. Under completion-only masking a truncated completion is a silently poisoned label.

Training details

Table

base	`google/gemma-3-4b-it`
method	LoRA r32 alpha64 on the language-model linears, completion-only masking
corpus	13,976 rows, 100% real SEC filings
epochs	3
max_len	2048, measured with the real tokenizer, not assumed
hardware	one A100-40GB, Spot

max_len is measured because two specs on this project were wrong about it and a truncated completion is a silently poisoned label. Batch size is 2 rather than the usual 4 for the same reason it has to be: at 2048 tokens, bsz 4 puts 8192 positions in a 262,145-wide loss tensor, which is the exact shape that ran this base model out of memory before.

Evaluation protocol

Base and tuned are generated in one process under identical greedy decoding (do_sample=False). Output length is compared base against tuned, and a tuned median below 0.6x the base median fails the run regardless of accuracy, which is a guard against buying a win rate with terseness. Every gate was proven able to fail by injecting deliberately corrupted rows before the corpus was trusted.

Usage

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "google/gemma-3-4b-it"
tok = AutoTokenizer.from_pretrained(BASE)
# AutoModelForCausalLM resolves gemma-3-4b-it to its multimodal wrapper, which is correct.
# Do NOT load it through a text-only class: the decoder is then randomly initialized and
# the model emits whitespace, with no error raised.
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype="bfloat16",
                                             attn_implementation="eager", device_map="cuda")
model = PeftModel.from_pretrained(model, "Jainamshahhh/flashfacts-4b").eval()

Try it, and everything that backs it

Live side by side demo: https://huggingface.co/spaces/Jainamshahhh/flashfacts-demo Enter your own input and watch the base model and FlashFacts-4B answer it under identical greedy decoding. The GPU backend scales to zero, so a cold first request takes about a minute.

Table with columns: Hugging Face, Kaggle
	Hugging Face	Kaggle
model	https://huggingface.co/Jainamshahhh/flashfacts-4b	https://www.kaggle.com/datasets/jainamshahhh/flashfacts-4b
dataset	https://huggingface.co/datasets/Jainamshahhh/flashfacts	https://www.kaggle.com/datasets/jainamshahhh/flashfacts-sec-earnings-extraction

Released on both platforms, with the scorer, every eval slice, the per row verdicts and significance.py alongside, so every number on this page can be recomputed rather than trusted.

Reproducibility and license

The released corpus, the scorer and every eval slice are published. Each row carries its SEC accession number and a SHA256 of the source exhibit, so any label can be traced to the exact document and rechecked. Apache-2.0, matching the base model. SEC filings are US government works in the public domain. Built with Adaptive Data by Adaption Labs.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Jainamshahhh

Model Tree

Base

google/gemma-3-4b-it

Adapter

this model

Input Modalities

Text

Output Modalities