Evaluation
The comprehensive suite contains 57 in-domain judgments over 25 unique
prompts (all 57 generation pairs freshly regenerated for the 7B) and 15
separate out-of-domain controls. Every judge received blind, position-randomized A/B
comparisons. The primary convention retains ties in the denominator; decisive
rates are reported only as sensitivity analyses.
Table with columns: Panel, Primary, ties included, Decisive sensitivity| Panel | Primary, ties included | Decisive sensitivity |
|---|
| 2025: GPT-4o + Gemini 2.5 Flash Lite | 93.0% | 95.5% |
| 2026: Claude Haiku 4.5 + Claude Sonnet 4.5 + GPT-5 + Gemini 3.1 Pro | 85.5% | 86.3% |
The previous higher decisive headline was a double-rounding artifact. The
exact mean of the decisive fractions rounds once to 95.5%.
Across the four 2026 judges, the prompt-level counts are 39/57 unanimous
Brie, 47/57 strict Brie majority, and 49/57 Brie plurality after
including two 2-Brie/1-base/1-tie vote patterns. The 49/57 figure is not a
strict-majority count.
The 2026 Anthropic judgments were routed through OpenRouter. Two earlier
direct-Anthropic files contain only API errors and are excluded from all
reported results.
Scaling boundary
The scaling result depends on the matched panel. On the shared 2025 GPT-4o and
Gemini panel, ties included, Qwen 0.5B / 3B / 7B score 78.9% / 93.9% / 93.0%.
On the exact shared 2026 Haiku, Sonnet, and GPT-5 subset, 7B leads 3B by 84.2%
to 80.1%. The evidence supports a large 0.5B-to-3B gain and a 7B lead under the
available 2026 overlap, not a universal size-performance ordering.
The n=15 out-of-domain control found performance near parity in aggregate, but
is underpowered for categorical claims about retained capability. Coding is
the clearest observed weakness.
Pass identities, raw counts, tie policy, and immutable hashes are documented
in docs/QWEN_7B_JUDGE_PROVENANCE.md in the paper repository.
Training details
- Base model:
Qwen/Qwen2.5-7B-Instruct
- Method: LoRA, rank 16, alpha 32, dropout 0.1
- Target modules:
q_proj, v_proj
- Training corpus: the full 1,213-row researcher-authored corpus, including
the 60 validation rows (see the paper's Section 7.2)
- Training: 1 epoch (152 steps); this published adapter is the final
checkpoint-152, promoted to the repository root on 2026-06-02 and loaded
as-is by the paper's 2026-06-14 evaluation — evaluated and published
weights are identical
- Language: English
Use
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base_id = "Qwen/Qwen2.5-7B-Instruct"adapter_id = "closestfriend/brie-v2-7b" tokenizer = AutoTokenizer.from_pretrained(base_id)base = AutoModelForCausalLM.from_pretrained(base_id, device_map="auto")model = PeftModel.from_pretrained(base, adapter_id)
This is a research artifact specializing one author's English-language
philosophical and creative register. Verify factual claims independently.
Resources
Citation
@misc{karman_brie_domain_adaptation, author = {Karman, Hunter N.}, title = {Human-Curated Data Authoring with LLMs: A Small-Data Approach to Domain Adaptation}, doi = {10.5281/zenodo.17657737}, url = {https://doi.org/10.5281/zenodo.17657737}}