gabeldev

deliverytwin-investigator-qwen3-1.7b-lora

README

License: apache-2.0

Results (DeliveryTwin benchmark, held-out final seed, n = 20 incidents per condition)

Exact-cause accuracy, with 95% Wilson intervals for the main rows. Paired comparisons use the exact McNemar test with no multiplicity correction. Each comparison is labelled pre-registered or exploratory in the project reports.

Table with columns: Condition, Qwen3-1.7B base, This adapter, Knowledge-based playbook
Condition	Qwen3-1.7B base	This adapter	Knowledge-based playbook
L0: full gauges, 3 fault families	0.05	0.45 [0.26, 0.66]	0.50
L1: L0 + telemetry noise	0.00	0.40 [0.22, 0.61]	0.70
L2: two causes with identical telemetry (needs an experiment)	0.00	0.90 [0.70, 0.97]	0.65
Same L2 incidents, experiment tool removed	—	0.45 [0.26, 0.66]	—
Same L2 incidents, one-experiment budget (L4 budgets)	0.00	0.80 [0.58, 0.92]	0.35
Same L2 incidents, imperfect twin lab (δ = 0.25)	0.00	0.95 [0.76, 0.99]	0.45
L5: L0 families with partial gauges (never seen in training)	0.00	0.40 [0.22, 0.61]	0.65

With vs. without the experiment tool on L2: 11 to 2, p = 0.022. This was pre-registered. Without experiments the model sits at the coin-flip ceiling for two observationally equivalent causes.
Adapter vs. base: p ≤ 0.022 on every condition.
On L5 the adapter never runs an experiment. API LLMs (gpt-5-mini, gpt-5.6-luna) reach 0.80–0.90 there by experimenting on their own (exploratory, p ≤ 0.022 against this adapter). The same API LLMs score 0.00–0.30 on the equivalent pair.

Known limitations — read before use

Distribution shortcut. Rejection-sampled expert traces teach which fault families each benchmark profile contains. Much of the L2/L4/L6 gain is consistent with the model having learned which probe discriminates in this profile, rather than general experimental reasoning. Its failure to experiment on L5 supports this. No ablation isolates it yet.
Catastrophic forgetting. On 150 fixed GSM8K test problems with greedy decoding, strict accuracy (final Answer: line) drops from 0.61 (base) to 0.14. A lenient last-number reading drops from 0.67 to 0.45; that reading still confounds format with arithmetic. The adapter mostly stops following output-format instructions. Use it only inside the DeliveryTwin protocol, not as a general assistant.
It imitates a single deterministic expert and inherits its blind spots. It also misreads gauges on L0/L1.
The data is 100% synthetic, and nothing indicates transfer to real operations. Any recommendation must go through human approval and an explicit constraint check before acting on anything real.
n is small (20 per condition), with one greedy run per model.

Training

Table with columns: Item, Value
Item	Value
Data	362 training and 132 validation investigation traces from a deterministic expert. Development seed only; disjoint from all evaluation seeds. Kept by rejection sampling against the evaluator, which uses the hidden truth only to keep or drop a trace and never appears in the text. Plus 90 replay examples from databricks-dolly-15k
Format	Qwen3 chat template in non-thinking mode (empty `<think>` block on every assistant turn), compact tool schemas, loss on assistant tokens only, sequences ≤ 4608 tokens
Hyperparameters	2 epochs, 113 optimizer steps, lr 2e-4 cosine, gradient accumulation 8, LoRA r = 16, α = 32, dropout 0.05, seed 0
Hardware	Single RTX 3050 Laptop (6 GB), 81 min, peak 3.05 GB
Validation loss (traces)	1.81 → 0.015

How to load

python

from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigimport torch base = AutoModelForCausalLM.from_pretrained(    "Qwen/Qwen3-1.7B",    quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",                                           bnb_4bit_compute_dtype=torch.bfloat16),    device_map="auto")model = PeftModel.from_pretrained(base, "gabeldev/deliverytwin-investigator-qwen3-1.7b-lora")tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")# Use apply_chat_template(..., tools=..., enable_thinking=False) with the DeliveryTwin# system prompt and tool schemas: deliverytwin/agent/llm_agent.py in the GitHub repo.# Full evaluation: scripts/run_g3.py --agent local --adapter <path> (see the repo README).

License and attribution

Adapter: Apache-2.0.
Base model: Qwen3-1.7B, Apache-2.0, Qwen Team / Alibaba Cloud.
Training replay used 90 examples of databricks-dolly-15k (CC BY-SA 3.0, Databricks). They are not redistributed here.
Evaluation used GSM8K (MIT, OpenAI).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

gabeldev

Model Tree

Base

Qwen/Qwen3-1.7B

Adapter

this model

Input Modalities

Text

Output Modalities