Usage
from transformers import AutoModelForCausalLM, AutoTokenizer repo = "aitos-io/Qwen3-0.6B-SmartHome-WeightEdit"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo) SYSTEM = "..." # the system prompt from the distil-labs/distil-home-assistant-qwen3 model card, unchangedTOOLS = [...] # the six tool definitions from the same model card, unchanged messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": "Is the front door locked?"}]text = tok.apply_chat_template(messages, tools=TOOLS, tokenize=False, add_generation_prompt=True, enable_thinking=False)inputs = tok(text, return_tensors="pt")out = model.generate(**inputs, max_new_tokens=128, do_sample=False)print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
Results
All numbers below were measured on the files in this repository, after merging and saving in the released precision.
Table with columns: Tool-call accuracy (95% CI), This model, Qwen/Qwen3-0.6B (unedited), distil-labs/distil-home-assistant-qwen3 (fine-tuned, reference)| Tool-call accuracy (95% CI) | This model | Qwen/Qwen3-0.6B (unedited) | distil-labs/distil-home-assistant-qwen3 (fine-tuned, reference) |
|---|
Official test set (distil-smart-home test.jsonl, n=50) | 84.0% (71.5%–91.7%) | 66.0% (52.2%–77.6%) | 92.0% (81.2%–96.8%) |
Official seed set (distil-smart-home train.jsonl, n=50) | 88.0% (76.2%–94.4%) | 68.0% (54.2%–79.2%) | 98.0% (89.5%–99.6%) |
| Independent held-out set C (n=120) | 83.3% (75.7%–88.9%) | 48.3% (39.6%–57.2%) | 90.0% (83.3%–94.2%) |
| Practice set B (n=120) | 90.0% (83.3%–94.2%) | 55.8% (46.9%–64.4%) | 98.3% (94.1%–99.5%) |
Table with columns: General ability, This model, Qwen/Qwen3-0.6B (unedited), distil-labs/distil-home-assistant-qwen3 (fine-tuned, reference)| General ability | This model | Qwen/Qwen3-0.6B (unedited) | distil-labs/distil-home-assistant-qwen3 (fine-tuned, reference) |
|---|
| ARC-Easy (200 test questions, letter answer) | 65.5% | 67.5% | 36.0% |
| WikiText-2 perplexity (40 paragraphs) | 47.59 | 46.43 | 48.16 |
The official test set was not used for any decision during editing; it was evaluated once, on the final model. Set C was written by an independent agent that could not see the editing code or the practice data. Intervals are Wilson 95% intervals; the official sets are small (50 items), so differences of a few points are within noise.
Honest comparison. The fine-tuned reference was trained (with gradients) on synthetic data for this exact task and is still more accurate. This model shows how far editing alone can take a general model, without any training.
How it was made
- Diagnosis. The unedited model fails mainly at a few decision tokens: the function name (e.g. answering "is the front door locked?" with
lock_door instead of get_device_status), refusals (intent_unclear) and output format. Linear probes show the user's intent is 99% decodable from middle layers while the output is right only 57% of the time on practice data — an obstacle, not a missing capability.
- Closed-form protected edit. For each of layers 18, 20, 22 and 24: teacher-force the gold answers on practice data, collect the MLP input activations at positions the model gets wrong (keys K) and the hidden-state shift that would make the gold token beat its strongest competitor (values R), then solve ΔW = Rᵀ K (KᵀK + λ·n·C)⁻¹, where C is the activation covariance of positions that must not change: already-correct answers (weighted 4×), prompt tokens, ordinary text (WikiText-2) and general question answering (ARC-Easy train, weighted 4×). Each layer's edit is applied at dose 1.2 before the next layer is solved.
- Gates. An edit is kept only if practice accuracy rises by ≥ 2 points, already-good categories do not drop, mean KL on ordinary text stays ≤ 0.03 and ARC-Easy drops ≤ 2 points.
- Version 2 (2026-11-12). Two changes from version 1: (a) the edited columns of each layer are chosen among neurons that are least active on ordinary text and general question answering (idea taken from earlier hand-written circuits, where repurposing near-idle neurons had no measurable side effects); (b) the practice data adds 300 cases written by an independent agent around the remaining errors — leaving out unspecified parameters, status questions that contain action words, and unsupported requests that look like supported ones. With side effects smaller, the dose was raised from 0.85 to 1.2.
- Evidence for the update, fixed in advance: on a fresh 500-case held-out set (set F, written by an independent agent and unseen during tuning), version 1 scores 72.6% and version 2 77.8%; paired comparison: 46 cases fixed, 20 broken, McNemar exact p = 0.0019. The unedited base scores 46.6% on the same set.
- Cost: ARC-Easy drops 2.0 points from the base model (version 1: 1.0), exactly at the 2-point gate; WikiText-2 perplexity rises 2.5%.
- Inspecting the edit with a logit lens shows the largest changes act as readable selection rules (e.g. a neuron that now pushes
get over lock, another that pushes heat over cool), added almost orthogonally to what those neurons did before. The effect is collective: keeping only the largest neurons of the edit loses most of the gain.
- Practice data: 240 template cases plus 600 diverse cases written by independent agents; dose and gates were checked on a separate 120-case practice set.
- The fine-tuned reference scores much lower on ARC-Easy here. Part of that gap is format: it was trained to always answer with a tool call, and this ARC check reads the answer letter from the first output token with no system prompt.
Limitations
- Covers only the six tools and the room / door / scene values of the distil-smart-home schema.
- Practice and held-out sets were generated by AI agents (templates and free writing), not collected from real users.
- Evaluation sets are small; treat point estimates with care.
- The edit was tuned for this task only; behaviour outside it was checked only via WikiText-2 perplexity and ARC-Easy.
License
Apache-2.0, same as the base model. Base model: Qwen/Qwen3-0.6B.