dean22029

pr_fm_qwen25_14b_adapter

README

License: apache-2.0

Results

Evaluated with one forward pass per pair (no generation), 4-bit NF4 base. The label base rate is ~50% by construction, so null log loss is 0.693 and accuracy is pair-winner accuracy. "zeroshot" is the same base model with no adapter, on the same prompts.

Table with columns: Split, Run, n pairs, Accuracy, AUC, Log loss, Δ vs null, Brier
Split	Run	n pairs	Accuracy	AUC	Log loss	Δ vs null	Brier
`test_chrono`	finetuned	58,910	0.598	0.633	0.6696	0.0235	0.2382
`test_chrono`	zeroshot	58,910	0.573	0.620	3.1876	-2.4945	0.3907
`test_within`	finetuned	61,120	0.654	0.716	0.6164	0.0767	0.2142
`test_within`	zeroshot	61,120	0.542	0.569	3.1059	-2.4127	0.4075

Two things to read carefully before quoting these numbers:

Fine-tuning mostly buys calibration, not discrimination. The zero-shot base models already rank pairs nearly as well; their log loss is far worse than the base-rate null because they are stuck near one letter. The adapter moves mean P(A) back to ~0.5.
Per-experiment results are highly heterogeneous. The overall test_chrono number is an average over 31 studies whose individual AUCs range from below chance to ~0.94. Rebuilding the evaluation (see Reproducing the data below) writes a by_experiment breakdown.

Training

Table

Base	`Qwen/Qwen2.5-14B-Instruct`
Method	QLoRA, 4-bit NF4 + double quant, bf16 compute
LoRA	r=16, alpha=32, dropout 0.05, on `q,k,v,o,gate,up,down_proj`
Optimiser	paged_adamw_8bit, lr 1e-4, cosine, warmup 3%
Batch	per-device 4, effective ~120 (A100 40GB, DDP)
Steps	6,000
Epochs completed	1.01
Max seq len	1024
Loss	masked to the assistant answer token only

The training set is 711,617 pairs. This run completed a full pass over it.

Validation choice accuracy over training: 0.632 @ step 2,000, 0.637 @ step 4,000, 0.647 @ step 6,000. It is flat over the last few evaluations, so the step limit is not obviously leaving accuracy on the table.

log_history.json in this repo has the step-by-step loss and eval metrics.

Input format

The adapter was trained on exactly this surface. Paraphrasing the system prompt or reordering the user block puts you out of distribution.

text

[system]You simulate survey respondents in conjoint experiments. Given the study context, the respondent's characteristics, and two options, predict which option the respondent chose. Answer with exactly one letter: A or B. [user]Conjoint study: <description>Country: <country>Year: <year> Respondent characteristics: <name: value; name: value; ...> The respondent chose between the following two options. Option A:- <factor>: <level>  ... Option B:- <factor>: <level>  ... Which option did the respondent choose?

The assistant target is a single letter, A or B.

Scoring convention

eval_example.py in this repo is a runnable version of all of this.

Do not generate. One forward pass; take logits[:, -1, :], index the single-token ids for "A" and "B", softmax over those two -> P(A).
Apply the chat template with add_generation_prompt=True, then tokenize with add_special_tokens=False — the template already added them.
Use padding_side="left" for batched scoring.
Randomise A/B orientation per pair, or average both orderings. Training data is orientation-randomised; a fixed order reintroduces position bias.
Load the base model first, then attach the adapter. adapter_config.json here points at Qwen/Qwen2.5-14B-Instruct, but keep the 4-bit config above to reproduce the numbers — loading the base in bf16 will shift them.

Splits and comparability

SPLITS.md and splits_summary.csv in this repo record exactly which experiments were trained on. The short version: training used every experiment with experiment_year <= 2021, so a test set built with a lower cutoff overlaps our training data. Check SPLITS.md before comparing these numbers against a model split a different way.

Reproducing the data

These adapters were trained on prompts built from the preference_fm conjoint data-share bundle (127 experiments; 115 usable). That bundle is not redistributed here and is not ours to redistribute — 113 of its 127 experiments carry no declared licence, and the underlying replication archives come from Dataverse and similar sources under their own individual terms. You need your own copy of the bundle to rebuild the data.

The scripts are bundled in pipeline/ in this repo.

Steps 1-3 are CPU-only. Step 1 needs R with jsonlite; steps 2-3 need tqdm and nothing else; step 4 needs the GPU stack from the example above.

bash

# 1. experiment.rds -> JSONRscript pipeline/export_experiments.R --zips <bundle>/experiment_zips --out exported # 2. reviewed covariate codebooks (optional; improves how X is verbalised)python pipeline/make_codebooks.py --exported exported --out codebooks # 3. prompts + splitspython pipeline/build_dataset.py --exported exported --out llm_dataset \    --cutoff-year 2021 --seed 42 # 4. score this adapterpython pipeline/evaluate_qlora.py --model <short-name> --adapter <this repo> \    --data llm_dataset/test_chrono.jsonl --out-prefix results/test_chrono

Step 3 must use --cutoff-year 2021 --seed 42 to reproduce our splits. Compare the splits_summary.csv it writes against the one in this repo: if they differ, the numbers on this card do not apply to your build. SPLITS.md explains what to check.

Note that the evaluated results on this card were produced without the codebooks of step 2 — every published number uses the raw column-dummy rendering of respondent covariates.

Licence

The adapter weights are a derivative of Qwen/Qwen2.5-14B-Instruct and inherit its terms (apache-2.0). The training data comes from third-party replication archives with their own individual licences and is not redistributed here.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

dean22029

Model Tree

Base

Qwen/Qwen2.5-14B-Instruct

Adapter

this model

Input Modalities

Text

Output Modalities