What it does
Input: one earnings press release (an 8-K's press-release exhibit, usually EX-99.1, with the exhibit that holds the income statement appended if the release lacks one) as Markdown, its tables as Markdown tables: the text column of the dataset's releases config, which secbench.inputs builds from the filing on EDGAR. Output: one JSON object in the reply format of the prompt extract_v2 (extract_v2.txt in this repository):
{"period_end": "YYYY-MM-DD", "quarter": {"revenue": {"value": ..., "unit": ...}, "net_income": {"value": ..., "unit": ...}, "eps_diluted": ...}, "year_to_date": {"revenue": {"value": ..., "unit": ...}, "net_income": {"value": ..., "unit": ...}, "eps_diluted": ...}}
quarter: the latest quarter the release reports; year_to_date: the fiscal year to date ending on period_end (the full fiscal year in a fourth-quarter release, the quarter again in a first-quarter release).
revenue: total revenue; net_income: net income attributable to the company; eps_diluted: diluted earnings per share. GAAP figures only; null for a figure the release doesn't report.
- Revenue and net income come as printed, with the table's unit (
"units", "thousands", "millions" or "billions"), and code converts them to dollars; EPS is a plain number in dollars per share.
The model is Qwen/Qwen3-8B at revision b968826d9c46dd6066d109eabc6255188de91218 with this adapter, thinking off (chat_template_kwargs {"enable_thinking": false}), at temperature 0.
How to use
Serve the base model with the adapter as every evaluation did, with vLLM 0.32.0 on Linux with an NVIDIA GPU (the test runs used one NVIDIA H100 NVL). vLLM 0.32.0's wheels are built for CUDA 13.0: pip installs them whatever the driver, but the server needs an NVIDIA driver for CUDA 13.0 or later.
pip install vllm==0.32.0vllm serve Qwen/Qwen3-8B --revision b968826d9c46dd6066d109eabc6255188de91218 \ --tokenizer-revision b968826d9c46dd6066d109eabc6255188de91218 \ --max-model-len 40960 --enable-lora --max-lora-rank 16 --max-loras 1 \ --max-cpu-loras 1 \ --lora-modules qwen3-8b-2k--step-00025=Jinnn2001/qwen3-8b-sec-earnings-lora
Then send each release as the benchmark's runner (secbench.extract) sent it: the prompt as the system message, the release as the user message, temperature 0, thinking off, at most 1,024 output tokens. With pip install httpx huggingface_hub, and a release saved as release.md (for example a text value of the dataset's releases config: load_dataset("Jinnn2001/sec-earnings-bench", "releases", split="company_test")):
import jsonfrom decimal import Decimal import httpxfrom huggingface_hub import hf_hub_download prompt_path = hf_hub_download( repo_id="Jinnn2001/qwen3-8b-sec-earnings-lora", filename="extract_v2.txt")with open(prompt_path, encoding="utf-8") as f: prompt = f.read() # the system message, verbatimwith open("release.md", encoding="utf-8") as f: release = f.read() # one press release as Markdown response = httpx.post( "http://localhost:8000/v1/chat/completions", json={ "model": "qwen3-8b-2k--step-00025", "messages": [ {"role": "system", "content": prompt}, {"role": "user", "content": release}, ], "temperature": 0.0, "chat_template_kwargs": {"enable_thinking": False}, "max_tokens": 1024, }, timeout=600.0,)response.raise_for_status()content = response.json()["choices"][0]["message"]["content"]reply = json.loads(content, parse_float=Decimal) # Revenue and net income come as printed, with their unit; EPS is a number.UNITS = { "units": 1, "thousands": 1_000, "millions": 1_000_000, "billions": 1_000_000_000,}figures = {"period_end": reply["period_end"]}for period in ("quarter", "year_to_date"): figures[period] = {} for name, figure in reply[period].items(): if isinstance(figure, dict): # {"value": 12345, "unit": "millions"} value, unit = figure["value"], figure.get("unit") figure = None if value is None else Decimal(value) * UNITS[unit] figures[period][name] = figureprint(json.dumps(figures, indent=2, default=str))
The server's context is 40,960 tokens and a reply may take 1,024 of them, so a prompt over 39,936 tokens doesn't fit: such releases (9 in the test sets) were left out of every evaluation, and the adapter is untested on them.
Results
From results/test_results.md (and its .json), the report of an evaluation protocol fixed before any test call (Evaluation protocol, below). Accuracy is the share of graded figures correct, a graded figure being one whose answer the release prints (not_in_input and no_answer figures are left out; see the dataset card), with 95% bootstrap intervals over companies.
Primary result (pre-registered): on the company test, step-00025 minus Qwen3-8B base: +10.12 points [+5.08, +15.58], from 89.12% to 99.24% of 662 figures; 73 discordant figures (70 fixed by training, 3 broken) from 16 companies.
Table with columns: model, runs on, company test, time test, cost / 1,000 releases, company / time test| model | runs on | company test | time test | cost / 1,000 releases, company / time test |
|---|
| step-00025 (this adapter) | 1× NVIDIA H100 NVL | 657/662 (99.24%) [98.05, 100.00] | 1,699/1,740 (97.64%) [96.64, 98.57] | 2.66(about0.56 without the instance freeze) / $0.59 |
| Qwen3-8B base | 1× NVIDIA H100 NVL | 590/662 (89.12%) [83.64, 94.19] | 1,516/1,740 (87.13%) [84.73, 89.49] | 0.54/0.55 |
| GPT-6.1 Sol | OpenAI API | 662/662 (100.00%) [100.00, 100.00] | 1,733/1,740 (99.60%) [99.14, 99.94] | 23.78/24.11 |
| Claude Haiku 5.5 | Anthropic API | 662/662 (100.00%) [100.00, 100.00] | 1,730/1,740 (99.43%) [98.91, 99.83] | 1.85/1.86 |
| GPT-6 Luna | OpenAI API | 654/662 (98.79%) [96.84, 100.00] | 1,731/1,740 (99.48%) [99.02, 99.88] | 1.23/1.24 |
- Company test: 148 releases from 37 companies held out of training; 662 graded figures.
- Time test: 346 releases from 344 of the training companies, filed 2026-07-01 to 2026-09-30, after every release drawn for training (filed 2023-10-10 to 2026-06-24); 1,740 graded figures. Its 317 releases with graded figures come from 315 companies, so the bootstrap there is in effect over releases.
- Cost: GPU runs at the price billed (2.538/hinthecompanytest′ssession,2.589/h in the time test's) times the run's wall time (first call to last; setup, model loading and idle time left out); API runs at their token prices. The GPU instance froze for 441 s during step-00025's company-test run: 2.66per1,000releaseswiththefreeze,about0.56 without it.
step-00025 minus each reference, in points, with 95% intervals:
Table with columns: step-00025 minus, company test, time test| step-00025 minus | company test | time test |
|---|
| Qwen3-8B base | +10.12 [+5.08, +15.58] | +10.52 [+8.15, +12.93] |
| GPT-6.1 Sol | −0.76 [−1.95, 0.00] | −1.95 [−2.94, −1.04] |
| Claude Haiku 5.5 | −0.76 [−1.95, 0.00] | −1.78 [−2.73, −0.91] |
| GPT-6 Luna | +0.45 [−1.41, +2.67] | −1.84 [−2.82, −0.96] |
On the time test step-00025 trails every API model, by 1.78 to 1.95 points, with 95% intervals below 0. On the company test it made 5 errors where GPT-6.1 Sol and Claude Haiku 5.5 made none (−0.76 points, interval −1.95 to 0.00), and it is level with GPT-6 Luna (+0.45, interval −1.41 to +2.67). Its case is cost, not accuracy: 0.59per1,000releasesonthetimetest,against1.24 to $24.11 for the API models.
step-00025's validation score, 98.0% (287/293), is chosen on validation (the checkpoint was picked for that score, so it may overstate; the test sets give the estimate).

Where it fails
Descriptive, not pre-registered: step-00025's misses on the test sets, grouped by the rule in results/test_results.md, which lists every one.
The common shareholders' line. Net income read from a line naming common shareholders, stockholders, shares or stock: income after preferred dividends or participating securities, where the answer key's NetIncomeLoss comes before them. Company test: 3 of its 5 misses, from 1 company (BA); 3 in releases whose filing reports preferred dividends and 0 in others, so dropping those releases, as below, drops all of them; 1 of the 3 figures training broke (the base model had them right). Time test: 29 of its 41 misses, from 16 companies (ALB, CMS, DOC, GOOGL, HPE, KDP, KIM, LHX, MCHP, NRG, PSA, SMCI, T, TDG, VMRK, WMB); 20 in releases whose filing reports preferred dividends and 9 in others (HPE, KDP, LHX print preferred-stock lines that the XBRL preferred-dividend check missed; DOC, TDG report income after participating securities), so dropping the preferred-dividend releases, as below, understates this failure on the time test; 11 of the 14 figures training broke (the base model had them right).
Preferred dividends. Accuracy without the releases whose filing reports preferred dividends (whole releases dropped: 11 on the company test, 20 on the time test), with 95% intervals:
Table with columns: model, company test, time test| model | company test | time test |
|---|
| step-00025 | 610/612 (99.67%) [98.95, 100.00] | 1,623/1,642 (98.84%) [98.08, 99.51] |
| Qwen3-8B base | 551/612 (90.03%) [84.19, 95.35] | 1,441/1,642 (87.76%) [85.29, 90.17] |
| GPT-6.1 Sol | 612/612 (100.00%) [100.00, 100.00] | 1,637/1,642 (99.70%) [99.28, 100.00] |
| Claude Haiku 5.5 | 612/612 (100.00%) [100.00, 100.00] | 1,637/1,642 (99.70%) [99.28, 100.00] |
| GPT-6 Luna | 604/612 (98.69%) [96.60, 100.00] | 1,636/1,642 (99.63%) [99.21, 99.94] |
Net income on those releases alone, counts only: company test (15 figures from 3 companies): step-00025 12/15, Qwen3-8B base 5/15, GPT-6.1 Sol 15/15, Claude Haiku 5.5 15/15, GPT-6 Luna 15/15; time test (28 figures from 15 companies): step-00025 8/28, Qwen3-8B base 12/28, GPT-6.1 Sol 28/28, Claude Haiku 5.5 25/28, GPT-6 Luna 27/28.
The quarter copied into year to date. In second- or third-quarter releases whose income statement has quarter columns only, with the year to date printed elsewhere, it repeated the quarter's figure as the year to date: WY 2 on the company test and TWLO 1 on the time test.
Other misses, by verdict: company test none; time test adjusted 3, noncontrolling_interest 1, wrong_line 4, wrong_period 3.
It never answered null. None of its predicted figures was null (0 of 888 on the company test, 0 of 2,076 on the time test); the API models' null figures, company test and time test: GPT-6.1 Sol 20 and 35, Claude Haiku 5.5 13 and 27, GPT-6 Luna 13 and 30. It answered every not_in_input figure (38 on the company test, 52 on the time test), XBRL answers the grader doesn't find printed. Some of those releases print the figure in another form, so answering isn't always wrong; but where a release doesn't report a figure the prompt asks for null, and no training label was null. Accuracy leaves these figures out.
Training
- Data. 3,200 releases of the training companies, labelled by code from the XBRL answer key: each figure as the release prints it (the value and its table's unit), never from a model's output. A release with any figure that couldn't be labelled that way was left out, so no label is null. 3,101 fit the 32,768-token cap (prompt and reply); 2,000 of them were drawn with seed 20261101, stratified by quarter kind (first quarter 581, second or third quarter 989, fourth quarter 430), filed 2023-10-10 to 2026-06-24. step-00025 had seen the first 400 of them, in training order.
- Method. LoRA r 16, alpha 32, dropout 0, on
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj and down_proj of every layer; the base model frozen in bfloat16, with gradient checkpointing; thinking off, each example laid out as vLLM sends it, with the loss on the reply's tokens only. AdamW (betas 0.9 and 0.999, eps 1e-08), peak learning rate 0.0002, a linear warmup over 10 steps, then a cosine decay; weight decay 0; gradient norm clipped at 1. One epoch of 125 steps of 16 examples (gradient accumulation), a checkpoint every 25 steps. Training loss (cross-entropy per reply token, train_log.jsonl): 0.12 at step 1, and 0.015 averaged over steps 1 to 25; a step's loss covers only its 16 examples, so it is noisy (from 0.00045 to 0.0088 over steps 21 to 25).
- Hardware. One Vast H100 NVL, in the session that also scored the checkpoints on validation.
- Checkpoint choice, on validation only. Every checkpoint was scored on the 60 validation releases of 10 companies (293 graded figures), with the base model for reference, never chosen; the rule: the most figures correct; ties go to the earliest step. step-00025's score is chosen on validation (the checkpoint was picked for that score, so it may overstate; the test sets give the estimate).
Table with columns: model, step, examples seen, correct / graded, revenue, net_income, eps_diluted| model | step | examples seen | correct / graded | revenue | net_income | eps_diluted |
|---|
| Qwen3-8B base | 0 | 0 | 262/293 (89.4%) | 85/102 | 89/102 | 88/89 |
| step-00025 (chosen) | 25 | 400 | 287/293 (98.0%) | 102/102 | 96/102 | 89/89 |
| step-00050 | 50 | 800 | 282/293 (96.2%) | 96/102 | 97/102 | 89/89 |
| step-00075 | 75 | 1,200 | 278/293 (94.9%) | 92/102 | 97/102 | 89/89 |
| step-00100 | 100 | 1,600 | 281/293 (95.9%) | 95/102 | 97/102 | 89/89 |
| step-00125 | 125 | 2,000 | 281/293 (95.9%) | 95/102 | 97/102 | 89/89 |
adapter_config.json is the checkpoint's own with two values changed: base_model_name_or_path named the training instance's local copy of the base model and is now Qwen/Qwen3-8B, and revision is now b968826d9c46dd6066d109eabc6255188de91218. Every other key is as saved.
Evaluation protocol
The analysis was fixed before any test call, in results/test_protocol.md (commit cf77acb, sha256 99767156…), with the company test's releases frozen by sha256. Primary result: on the company test, step-00025's accuracy minus Qwen3-8B base's over all kept releases, with a 95% bootstrap interval over companies (10,000 draws, numpy default_rng(20261105)); everything else is descriptive. No checkpoint or setting was chosen on a test set.
Deviations, as results/test_results.md records them (the section names in parentheses are that report's):
No amendment was made (none are allowed after the first test call); these are recorded here instead.
- The time test. The protocol set its cutoff for when the Q3 2026 releases are added, and said the provisional 346-release time test would not be scored. Decision 2026-11-07 (commit e352c14) froze that same set (releases filed 2026-07-01 to 2026-09-30) at 12:20 UTC on 2026-11-07: after the company-test runs (11:16-11:31 UTC) and before any time-test call (12:32 UTC). No amendment records it, and none were allowed. Every time-test result was descriptive under the protocol anyway.
- The company test's GPU price. The protocol prices GPU runs at the /hgiventovastsetup.sh(run.json′sgpuhourlyusd).Bothcompany−testGPUrunsrecord2.776,anexamplepricepassedtovastsetup.shbymistake,whichisthepricebytheprotocol′sletter;theinstancewasbilled2.538/h, which the costs here use (both GPU runs' costs at the recorded price are also given, under Cost and latency). The time test's price, $2.589/h, was recorded correctly.
- GPT-6.1 Sol's time-test rerun ran 2 calls at a time, not the protocol's 4 (Cost and latency): its first attempt hit the 1,000,000 tokens-per-minute limit at 4 (Time test, Coverage).
- The bootstrap's details were fixed on 2026-11-07, after the test runs were graded (Bootstrap, with its robustness line).
Serving
From the serving benchmark (results/serving_results.md): one NVIDIA H100 NVL at $3.429 per GPU-hour, vLLM 0.32.0, prefix caching off, --max-model-len 40,960; vLLM's defaults otherwise. Load test: the time test's prompts, 128 per level after 8 warm-up requests, at 1, 2, 4, 8, 16, 32 and 64 requests in flight; the best level is the one with the most releases per hour, and a cost per 1,000 releases is the GPU's price for the level's makespan. Same figures as adapter: the validation figures (6 per release) equal to the adapter variant's. Validation is where step-00025 was chosen, so every variant's validation score may overstate: it compares the variants, and the test results above give the level. The int4-awq model is on the Hub as Jinnn2001/qwen3-8b-sec-earnings-int4-awq.
Table with columns: variant, serves, status, weights (GiB), KV cache (tokens), best releases/h (in flight), latency p95 there (s), /1,000releasesat3.429/h, at $1/h, validation correct (step-00025 chosen on it), same figures as adapter| variant | serves | status | weights (GiB) | KV cache (tokens) | best releases/h (in flight) | latency p95 there (s) | /1,000releasesat3.429/h | at $1/h | validation correct (step-00025 chosen on it) | same figures as adapter |
|---|
| adapter | the base model in bf16 with the LoRA adapter, as stage 2 served it but with prefix caching off | complete | 15.36 | 486,720 | 4,491 (32) | 41.09 | $0.763 | $0.223 | 287/293 (98.0%) | (itself) |
| merged | the adapter merged into the base weights, bf16 | complete | 15.27 | 489,952 | 4,945 (32) | 37.10 | $0.693 | $0.202 | 287/293 (98.0%) | 360/360 figures, 60/60 releases |
| fp8 | merged, quantized to FP8 by vLLM as it loads | complete | 9.07 | 535,088 | 6,143 (32) | 28.43 | $0.558 | $0.163 | 285/293 (97.3%) | 358/360 figures, 59/60 releases |
| int4-awq | merged, 4-bit weights with AWQ | complete | 5.78 | 558,608 | 4,938 (32) | 36.38 | $0.694 | $0.202 | 280/293 (95.6%) | 353/360 figures, 56/60 releases |
Limitations
- Behind the API models on the time test, by 1.78 to 1.95 points; its case is cost.
- The common shareholders' line: 29 of its 41 time-test misses are net income read from a line after preferred dividends or participating securities (Where it fails). Check net income where a release prints such a line.
- No nulls in training: a release was labelled only when every figure it should report is printed, so no training label is null, and the adapter never answered null on the test sets.
- Answer-key gaps: no XBRL answer for 124 of the 1,916 figures the time test's releases should report (6.5%), and for 28 of 728 on the company test (3.8%); those figures aren't graded.
- Fourth-quarter EPS isn't graded: XBRL has no fourth-quarter value, and EPS can't be derived by subtraction.
- One training run, untuned: its settings were fixed before training, and the chosen checkpoint is the first one saved (step 25 of 125).
- Survivorship: the companies are the S&P 500's members on one date, financial companies excluded, so companies that have left the index are absent.
- No Q3 2026 releases in the time test: it holds releases filed 2026-07-01 to 2026-09-30; its cutoff was set before all the Q3 2026 10-Qs, which give the answer key, were filed (a deviation from the protocol).
- Long releases: prompts over 39,936 tokens were left out (9 test releases), so the adapter is untested on them.
- Scope: English press releases of US companies reporting under US GAAP, read with the prompt it was trained on; other documents, languages, accounting standards or prompts are untested.
- Not financial advice: a figure can be wrong; check it against the filing before relying on it.
License
Apache-2.0, like Qwen/Qwen3-8B. The benchmark's code is MIT-licensed (GitHub); the dataset card states its data's terms.
Citation
@misc{sankaran2026secearningsbench, author = {Swaminathan Sankaran}, title = {{sec-earnings-bench}: extracting {GAAP} figures from {SEC} earnings releases, graded against {XBRL}}, year = {2026}, url = {https://github.com/jinnn11/sec-earnings-bench}}
Links