greg1506

qwen3-8b-coffee-v10

README

License: apache-2.0

Intended use

A local, self-contained coffee knowledge assistant: answer factoid barista/roasting/method questions, define terms, and do standard-dose brew calculations in the types it was trained on. Single-language (English), corpus-bounded domain expert.

Performance (CoffeeBench v6, 336 items, held-out; raw exact match, style-robust soft EM)

Table with columns: Family, Raw EM, Soft EM
Family	Raw EM	Soft EM
roast-map	1.00	1.00
quality-standards	1.00	1.00
legacy-core	1.00	1.00
ratio-forward	0.98	0.98
solids-mass	0.93	0.93
species-origin	0.87	0.93
ratio-inverse	0.85	0.85
ey-inverse	0.74	0.74
method-map	0.75	0.88
definition	0.67	0.83
milk-drinks	0.62	0.88
classification	0.67	0.67
ey-forward	0.60	0.62
calculation	0.00	0.00
overall	0.753	0.789

No memorization split: items far from the training distribution score at or above near-distribution items (0.80 vs 0.78 soft EM).

Known limits

Forward arithmetic is unreliable (~60%): extraction-yield forward calculations with decimals fail silently while sounding confident. Human-verify every numeric answer, or keep calculations in the trained regime (ratio-inverse / solids-mass / ey-inverse are strong).
Processing taxonomy has real gaps (Kenyan double ferment, pulped natural, honey-process mucilage), n=9 so it does not dominate the headline, but treat answers as unverified.
Bench is synthetic and built from the same corpus the model trained on (leak-free verbatim, same distribution); no blind expert assessment yet — treat scores as self-referential.
Out-of-domain questions are unverified; this model is not a general assistant.

Recipe (abridged)

Domain token stream (109M coffee-weighted tokens) → LoRA continued-pretraining-style pass → merged base
Synthetic instruction SFT: brew-ratio / extraction-yield quotient grids, bare-numeric answers, held-out bench combos
Teacher-synthetic Q/A mined from 6k corpus train docs with a local Qwen3-8B vLLM server (numeric-free pairs only) → low-LR (3e-5) top-up on the v2 release

Later top-ups (a second mined round, then a third) did not improve the score (0.753 → tie → 0.714), so v10 is the milestone release.

Version history

Table with columns: Release, Raw EM, Soft EM, Note
Release	Raw EM	Soft EM	Note
coffee-v2	0.735	—	domain base + v6 instruct adapter
coffee-v10 (this)	0.753	0.789	+ LLM-mined numeric-free QA top-up

Training details

Base: Qwen3-8B (Apache-2.0), LoRA r=32/alpha=64, bf16, seq 512, effective batch 32
Final top-up: lr 3e-5, 3 epochs (early stop on eval plateau), ~10.7k instruction records
Eval: exact match + token F1 + style-robust soft EM on 336 held-out items
Data licensing: mixed open corpora; corpus card and per-document licenses in the source repo (coffee-corpus v2.0)

Disclaimer

Not a source of record: independently verify numeric dosing/extraction calculations before using them.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

greg1506

Model Tree

Base

Qwen/Qwen3-8B

Fine-tuned

this model

Input Modalities

Text

Output Modalities