Evaluation
The comprehensive evaluation contains 57 blind, position-randomized A/B
judgments over 25 unique prompts (none in the training data). Primary rates
retain ties and all assigned prompts.
Table with columns: Result, Brie preference, Construction| Result | Brie preference | Construction |
|---|
| 2025 primary | 67.8% | Mean of three full n=57 passes: Claude Opus 4, GPT-4o, and Gemini 2.5 Flash Lite |
| Expanded sensitivity | 69.9% | Adds Claude 3.5 Sonnet (n=42); prompt coverage is unequal |
| Historical split-source result | 71.9% | Original disjoint Claude pool, 41/57 |
One record in the historical split source was stored as unknown because its
judgment said Winner: Neither rather than the requested A/B/Tie token. It is
a semantic tie and remains in the denominator. Deleting it produces the older
73.2% parser-exclusion figure; averaging that parser-exclusion source with GPT
and Gemini produces 77.0%. Neither is the preferred headline.
The three full-pass inputs to the 67.8% result are Claude Opus 4 at 26/57,
GPT-4o at 43/57, and Gemini 2.5 Flash Lite at 47/57. The separate n=13
in-domain and n=15 out-of-domain suites are not merged into this comprehensive
panel.
Complete pass identities, hashes, and reporting decisions are documented in
docs/QWEN_0_5B_JUDGE_PROVENANCE.md in the paper repository.
Training details
- Base model:
Qwen/Qwen2.5-0.5B-Instruct
- Method: LoRA, rank 16, alpha 32, dropout 0.05 (earlier cards stated rank 8
/ alpha 16; the shipped
adapter_config.json is authoritative)
- Target modules:
q_proj, v_proj
- Training corpus: the 1,153-example snapshot of the researcher-authored
corpus (see the paper's Section 7.2)
- Training: configured for 2 epochs (290 steps) with best-checkpoint
selection; this published adapter is checkpoint-200 (epoch ≈1.38), the
best-validation-loss checkpoint, and is the adapter the paper evaluated
- Language: English
Use
from transformers import AutoModelForCausalLM, AutoTokenizerfrom peft import PeftModel base_id = "Qwen/Qwen2.5-0.5B-Instruct"adapter_id = "closestfriend/brie-qwen2.5-0.5b" tokenizer = AutoTokenizer.from_pretrained(base_id)base = AutoModelForCausalLM.from_pretrained(base_id, device_map="auto")model = PeftModel.from_pretrained(base, adapter_id)
This is a small research model with substantial capacity limits. Its training
corpus reflects one author's English-language philosophical and creative
practice; verify factual or high-stakes output independently.
Resources
Citation
@misc{karman_brie_domain_adaptation, author = {Karman, Hunter N.}, title = {Human-Curated Data Authoring with LLMs: A Small-Data Approach to Domain Adaptation}, doi = {10.5281/zenodo.17657737}, url = {https://doi.org/10.5281/zenodo.17657737}}