Highlights (held-out, vs the original Qwen3.5-0.8B)
Table with columns: Capability, Original, Qwen3.5-0.8B-NKC| Capability | Original | Qwen3.5-0.8B-NKC |
|---|
| MATH-500, thinking mode | 6% | 53% |
| GSM8K, thinking mode | 16% | 76% |
| IFEval (strict prompt-level), thinking mode | 20% | 62% |
| Push-T: "will the pusher move the block?" | 50% (chance) | 88-89% |
| Depth: "which point is farther from the camera?" (NYU) | 75.3% | 82.6% |
What was changed
Table with columns: Tensor(s), Entries changed, Edit, Knowledge source| Tensor(s) | Entries changed | Edit | Knowledge source |
|---|
model.language_model.layers.15.mlp.down_proj.weight, ...layers.19.mlp.down_proj.weight | 130,188 | removal of a "reasoning loop" direction from 64 output columns per layer | published edit RikiyaT/Qwen3.5-0.8B-weight-space-programming (run p50c), applied bit-exactly |
model.visual.merger.linear_fc2.weight / .bias | 3,146,752 | rank-4 update = depth channel (rank 1) + physics channel (rank 3) | Depth Anything V2 Small; JEPA-WM (Push-T world model) |
Total: about 3.28M of 873.4M weights changed (0.38%). Every changed tensor, its source and hashes are recorded in
nkc_manifest.json.
Methodology
Recipient anatomy. Qwen3.5-0.8B = ViT (12 blocks) -> patch merger (LayerNorm -> 3072 -> GELU -> linear_fc2 -> 1024)
-> 24-layer hybrid language model. All visual edits are low-rank updates of linear_fc2:
merger_out' = W2 h + b2 + U (R h + r0), i.e. W2 += U R, b2 += U r0 (no new parameters).
1. Reasoning-loop repair (published, reproduced). With thinking on, the base model loops until the 32,768-token cap
on most problems. The published p50c edit subtracts the loop direction from the MLP output columns that write it most
(layers 15 and 19). We verified by tensor hashes that our base is the exact revision the edit was made on
(Qwen/Qwen3.5-0.8B@2fc06364) and applied it with the authors' own apply_edit.py (hash-checked).
2. Depth (amplifying an existing ability, training-free). Depth Anything V2 depth maps on unlabeled COCO images are
read out of Qwen's own 3,072 merger neurons with closed-form ridge regression (held-out R^2 = 0.36). The readout is
written along the language model's own "near minus far" token-embedding direction; only its strength is chosen on dev
data. No gradient steps.
3. Push-T physics (introducing a new ability, small calibration). JEPA-WM's predicted latent change ("will this
push move the block?") supervises the edit. The readout is an interference-regularised generalised ridge:
w = (C_task + mu C_photos + lam I)^-1 c_task,donor, with mu chosen so its response on natural photos is <= 1% of its
response on Push-T scenes. That stops the physics edit from overwriting depth. The write map (3 x 1024 = 3,072 scalars)
was calibrated in 300 Adam steps on 2,400 scenes labelled only by the world model (no simulator ground truth).
What did not work (and is not in this model). DINOv2 object-grouping transfer (no held-out gain, damaged depth);
a null-space projection against natural-image activations (did not prevent interference); our own extra loop-removal
edit (worse on held-out math).
Evaluation protocol
- Thinking-mode text (researchers' protocol): vLLM, bf16, chat template with thinking on, prompt = problem +
"Please reason step by step, and put your final answer within \boxed{}.", temperature 0.6, top_p 0.95, top_k 20,
presence_penalty 1.5, max 32,768 new tokens, seed 0; math graded with math-verify, IFEval with the official checkers
on the answer after
</think>. Fixed seeded subsets: MATH-500 100, GSM8K 50, IFEval 50 problems.
- Default (non-thinking) text: model-card non-thinking sampling (T 0.7, top_p 0.8, top_k 20, presence 1.5, 8,192
tokens), FULL MATH-500 (500), GSM8K test (1,319), IFEval (541).
- Vision: two-choice letter scoring (next-token logit A vs B). NYU Depth V2 test (580 point pairs, 145 images),
Push-T validation episodes never used for calibration (490 items, simulator ground truth), COCO localisation built
from human box annotations (321 items, images never used for any edit), BLINK val subsets, COCO caption likelihood.
- Paired 95% CIs by bootstrap clustered by image / episode. Text-only behaviour verified byte-identical to the
published math model (greedy tokens identical, max logit difference 0.0).
Full results
Text
Table with columns: Benchmark, Original, NKC| Benchmark | Original | NKC |
|---|
| MATH-500, thinking (100) | 6% | 53% |
| GSM8K, thinking (50) | 16% | 76% |
| IFEval, thinking (50) | 20% | 62% |
| Share of answers cut at 32k tokens (MATH-500 / GSM8K / IFEval) | 96% / 82% / 78% | 32% / 2% / 2% |
| MATH-500, default mode (500) | 46.6% | 44.4% |
| GSM8K, default mode (1,319) | 57.7% | 55.4% |
| IFEval, default mode (541) | 53.0% | 48.8% |
For reference, the published p50c evaluation (4 / 2 / 2 samples on the full sets) reports 7.9 -> 48.1 (MATH-500),
13.8 -> 67.1 (GSM8K), 21.0 -> 62.3 (IFEval).
Vision (paired difference vs original, 95% CI)
Table with columns: Benchmark, Original, NKC, Delta| Benchmark | Original | NKC | Delta |
|---|
| NYU depth "closer" | 77.1% | 82.2% | +5.2 [+2.4, +7.9] |
| NYU depth "farther" | 75.3% | 82.6% | +7.2 [+3.8, +10.9] |
| NYU depth, reworded prompt | 84.0% | 86.2% | +2.2 [-0.3, +4.8] |
| Push-T "will it move" | 50.0% | 88.4% | +38.4 [+34.9, +41.7] |
| Push-T "will it stay" | 50.0% | 89.2% | +39.2 [+36.1, +42.4] |
| Push-T, reworded (never used in calibration) | 50.0% | 71.8% | +21.8 [+16.5, +26.9] |
| BLINK relative depth | 64.5% | 66.9% | +2.4 |
| BLINK spatial relation | 62.2% | 62.9% | +0.7 |
| BLINK counting | 50.0% | 54.2% | +4.2 |
| BLINK object localisation | 66.4% | 53.3% | -13.1 |
| COCO localisation (321) | 81.3% | 77.3% | -4.0 |
| COCO caption NLL (lower = better) | 2.83 | 2.91 | +0.08 |
Counterfactual physics test. Same frame, only the push changed (rotated 0/90/180/270 degrees), ground truth from the
Push-T physics simulator (fidelity check: 0.009 px median error on recorded actions). On scenes where some pushes move
the block and others do not, NKC ranks a moving push above a static one 90-94% of the time (original: 48-62%; a
model that only judged the scene would score 50%). Its prediction follows the action.
Controls and causal tests
Table with columns: Test, Result| Test | Result |
|---|
| Depth edit with random readout / Qwen's own PCA readout (same write) | 74.3/72.8 and 76.0/71.2 vs donor 80.7/86.0 (closer/farther) |
| Depth: negate / zero / shuffle the transferred feature | 55.9% / 77.1% (= baseline) / 75.7% |
| Depth: readout fit on real sensor depth instead of the donor | 81.2/81.9: the donor matches, not beats, ground-truth supervision |
| Physics: random / PCA readout + same calibration | 87.3/84.9 and 89.2/84.3: the gain comes from the donor's supervision, not its internal features |
| Physics: negate / zero / shuffle the feature | 24.9% (inverted) / 50.0% / 53.3% |
| Physics without the interference-regularised readout | depth "farther" falls to 57.9% (with it: 81.4%) |
Limitations
- The math gain is a repair of thinking-mode looping; it does not add mathematical knowledge. In default mode the edit
costs 2-4 points. Use
enable_thinking=True for math.
- Object localisation and caption likelihood regress. The visual edits sit in the patch merger, which never sees the
question, so they reach every image task.
- The physics ability is a narrow, learned move/stay decision for Push-T-style scenes (reworded prompt: 71.8%), not
general physical reasoning.
- Thinking-mode subsets are small (50-100 problems; about +/-5-7 points of sampling noise).
Usage
from transformers import AutoModelForImageTextToText, AutoProcessor model = AutoModelForImageTextToText.from_pretrained("MichielH/Qwen3.5-0.8B-NKC", dtype="bfloat16", device_map="auto")processor = AutoProcessor.from_pretrained("MichielH/Qwen3.5-0.8B-NKC") messages = [{"role": "user", "content": [{"type": "text", "text": "What is 17 * 23? Please reason step by step, " "and put your final answer within \\boxed{}."}]}]inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", enable_thinking=True).to(model.device)out = model.generate(**inputs, max_new_tokens=4096, do_sample=True, temperature=0.6, top_p=0.95, top_k=20)print(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Recommended for reasoning: thinking on, temperature 0.6, top_p 0.95, top_k 20, presence_penalty 1.5 (vLLM).
Credits
- Base model: Qwen team, Qwen/Qwen3.5-0.8B (Apache-2.0).
- Reasoning edit: RikiyaT et al., "Sparks of Weight-Space Programming" (exp8_math, run p50c),
model and
logs.
- Depth donor: Depth Anything V2 Small. Physics donor: JEPA-WM (Push-T checkpoint).