1. What the two donors actually changed (measured, fusion/delta_probe.py, fusion/gauge_probe.py)
- Nemotron = base + a tiny, sparse update: ‖τ_N‖/‖W‖ ≈ 0.1–0.4 % per matrix, 30–45 % of bf16
entries touched, norms / vision / audio untouched.
- QAT looks like a huge change (‖τ_Q‖/‖W‖ ≈ 15–55 %), but it is mostly gauge: each matrix is ≈ a
scalar multiple of the base matrix (cos ≈ 0.993–0.996;
down_proj ≈ 0.46×, k_proj ≈ 0.74×,
per_layer_model_projection ≈ 0.0154×), with the scale absorbed by the RMSNorm that follows. The
remaining ~10 % is the Q4_0 lattice: every 32-block has ≤ 16 distinct values d·n, n ∈ {−8…7}
(12.4 on average vs 31.6 for the base). The "unquantized" release is the 4-bit model, written in bf16.
- The two task vectors are orthogonal (cos ≈ 0.000, sign agreement 0.50).
- QAT is not a fixed point of llama.cpp's own Q4_0 rounding (5 % relative change): Google's block
scales were learned, so re-quantizing the bf16 QAT file with
llama-quantize Q4_0 does not reproduce
Google's 4-bit model. fusion/harness.py:recover_grid recovers the per-block scales (0.2 % error = bf16
storage rounding).
2. The merge: gauge-transported task arithmetic
Naive task arithmetic W_base + τ_Q + τ_N adds Nemotron's update, which is expressed in the base's
parameterization, onto matrices QAT has rescaled by 0.46–0.88× — silently re-weighting the RL update by
1.1–2.2× depending on the layer. Instead the Nemotron delta is transported into QAT's gauge first:
s = <W_base, W_qat> / <W_base, W_base> (per tensor; per row for the two embedding tables)W_new = W_qat + λ · s ⊙ (W_nemo − W_base)
λ (the strength of Nemotron's transported update) is chosen on practice splits only (fusion/screen.py).
The 54 k_proj / v_proj / k_norm tensors of the KV-shared layers 24–41 — which the forward pass never
reads (NVIDIA ships them in a separate "frozen" file; Google's QAT release drops them) — are dropped.
Vision/audio towers: Nemotron never touched them (bit-identical to base, checked tensor-by-tensor). QAT changed
exactly one: embed_vision.embedding_projection (≈0.86× base, re-gauged to QAT's rescaled embedding space). The
merge rule gives Q + s·(N−B) = Q there, so the merged model uses QAT's projection.
Chosen λ = 1 (practice data only; results/screen.jsonl, results/lambda_choice.json):
Table with columns: λ, MC practice (4 sets), synthetic IF, DEV n=192, WikiText ppl| λ | MC practice (4 sets) | synthetic IF, DEV n=192 | WikiText ppl |
|---|
| 0 (= QAT) | 77.2 | 88.0 | 31.3 |
| 1 | 77.9 | 96.4 | 42.4 |
| 1.5 | 77.6 | 94.8 | 49.2 |
λ=1 vs λ=0 on IF‑DEV: +8.3 pp (SE 2.0), 16 prompts fixed, 0 broken — Nemotron's update survives transport
onto QAT's lattice. Extrapolation (λ>1) did not help.
Carried over from Nemotron beyond the weights: its chat template (conversational_mode, assistant content
preserved before tool calls, Gemma-4-matching generation prompt when thinking is off) and tokenizer config.
3. Weight-space programming (after Takehi et al., "Sparks of Weight-Space Programming", Exp. 8)
The blog's working edit removed a failure direction from residual-writing matrices,
W' = W − a·u(uᵀW), with u found from recorded forward passes and every edit screened on a DEV
split, confirmed on a HOLDOUT split and compared with a random-direction control. Same protocol here
(fusion/wsp.py, fusion/wsp_run.py, fusion/wsp_screen.py):
- practice data only (GSM8K train; a synthetic instruction-following set built from ARC-train content with
IFEval-registry constraints and the official checkers — disjoint from IFEval's test prompts);
- three disjoint splits fixed before any edit: PROBE (find directions), DEV (screen), HOLDOUT (confirm once);
- forward passes: generation passes label each answer, then teacher-forced passes record what every
layer's
o_proj and down_proj writes (84 sites), averaged over answer tokens;
u = normalized difference of failure and success means at a site; sites ranked by effect size d′;
- candidates = top-k sites × strength a; the winner on DEV is confirmed on HOLDOUT against the unedited
model and against norm-matched random directions at the same sites and strength.
Every forward pass is logged in results/wsp/passes.jsonl (5,264 sequences, 758k tokens for this phase).
Outcome: no edit is shipped. Directions were ranked by 2-fold cross-validated d′ (in-sample d′ is badly
inflated in 2560 dims). Paired DEV results vs the unedited merge:
Table with columns: candidate, Δ (pp), SE, fixed / broken| candidate | Δ (pp) | SE | fixed / broken |
|---|
| GSM8K top-2, a=1.0 | −1.0 | 1.8 | 5 / 7 |
| GSM8K top-4, a=0.5 | +0.5 | 1.9 | 7 / 6 |
| GSM8K top-8, a=0.5 | −1.0 | 2.3 | 9 / 11 |
| GSM8K top-8, a=1.0 | −0.5 | 2.6 | 12 / 13 |
| IF top-2, a=1.0 | −1.0 | 1.0 | 1 / 3 |
| IF top-4, a=0.5 | −1.6 | 1.2 | 1 / 4 |
Nothing beat the unedited model on DEV, so HOLDOUT was never touched (results/wsp/wsp_decision.json). This
matches the blog's own Gemma-3-270M follow-up: rank-one removal works when one failure mechanism dominates
(Qwen3.5's looping, 93 % of answers at the token limit); here only ~8 % of GSM8K answers hit the limit and the
failure signal is spread across almost every layer (cross-validated d′ ≈ 1.2 at nearly all 84 sites).
4. Folding the encoders into one stack (fusion/fold.py)
Gemma-4-12B is encoder-free because Google trained it with linear patch / waveform embedders. A 16-layer
ViT and a 12-layer conformer cannot be turned into a linear map algebraically, so this does not claim the
12B's FLOP savings. What is done, exactly:
- Patch-embedder affine fold –
input_proj(2(x−0.5)) = (2W)x − W·1; −W·1 moves into the x-axis
position table → a pure linear patch embedding + 2-D position table (the 12B vision_embedder form).
- Pooler-scale fold –
RMSNorm(c·h; ε) = RMSNorm(h; ε/c²): the √768 pooler scale disappears into the
bridge norm's ε.
- Subspace embedding – a width-d Gemma block acting on
[x_d, 0] in the 2560-wide stream equals the
block with zero-padded weights and RMSNorm gain·√(d/D), ε·d/D. All 28 encoder layers become blocks of the
same 2560-wide residual stack (vision reads/writes a 768-dim subspace, audio a 1024-dim one).
Result: one ordered list of 70 blocks (16 vision-bidirectional, 12 audio-conformer, 42 text-causal) + two
bridge blocks, one checkpoint, one forward. Verified on the final weights (results/fold_verification.json):
image soft tokens rel. err 7×10⁻⁷ (fp64), audio 5×10⁻⁸, zero-padded block 8×10⁻⁸, end-to-end logits with
identical image features 0.0.
5. Benchmarks (held-out test splits, thinking off, greedy, same prompts/limits for every model)
Final model = gauge-transported merge at λ=1, no edits. Δ with paired-bootstrap 95 % CI vs Nemotron:
Table with columns: benchmark, Nemotron‑3.5‑Bolt, this model, Δ [95 % CI]| benchmark | Nemotron‑3.5‑Bolt | this model | Δ [95 % CI] |
|---|
| MMLU (570) | 71.4 | 70.4 | −1.1 [−3.0, +0.9] |
| ARC‑Challenge (400) | 90.2 | 89.5 | −0.8 [−2.8, +1.0] |
| HellaSwag (400) | 81.5 | 81.2 | −0.2 [−2.5, +2.0] |
| WinoGrande (400) | 65.5 | 64.8 | −0.8 [−3.5, +2.0] |
| TruthfulQA mc1 (300) | 75.3 | 74.7 | −0.7 [−3.3, +1.7] |
| Global‑MMLU‑Lite, 8 langs (400) | 58.2 | 55.2 | −3.0 [−5.8, −0.2] |
| GSM8K (250) | 67.2 | 68.4 | +1.2 [−2.4, +4.8] |
| MATH‑500 subset, lenient (120) | 55.8 | 59.2 | +3.3 [−3.3, +10.0] |
| IFEval prompt‑strict (200) | 85.0 | 85.0 | 0.0 [−3.5, +3.5] |
| HumanEval pass@1 (164) | 85.4 | 86.0 | +0.6 [−2.4, +3.7] |
| WikiText‑2 ppl (lower = better) | 149.7 | 50.4 | |
Three-way, on identical items (QAT generation suites ran on the first 120 / 80 items; all three models are
compared on that same prefix — results/three_way.md):
Table with columns: benchmark (n), QAT, merge, Nemotron, merge − QAT [95 % CI], merge − Nemotron [95 % CI]| benchmark (n) | QAT | merge | Nemotron | merge − QAT [95 % CI] | merge − Nemotron [95 % CI] |
|---|
| TruthfulQA (300) | 68.3 | 74.7 | 75.3 | +6.3 [+3.7, +9.3] | −0.7 [−3.3, +1.7] |
| MATH‑500, lenient (80) | 48.8 | 60.0 | 55.0 | +11.2 [+1.2, +21.2] | +5.0 [−3.8, +13.8] |
| GSM8K (120) | 62.5 | 66.7 | 65.0 | +4.2 [−0.8, +9.2] | +1.7 [−1.7, +5.0] |
| HumanEval (120) | 85.8 | 89.2 | 90.0 | +3.3 [−0.8, +8.3] | −0.8 [−4.2, +2.5] |
| IFEval strict (120) | 80.8 | 83.3 | 84.2 | +2.5 [−5.0, +10.8] | −0.8 [−5.0, +3.3] |
| ARC‑Challenge (400) | 88.5 | 89.5 | 90.2 | +1.0 [−0.5, +2.5] | −0.8 [−2.8, +1.0] |
| WinoGrande (400) | 63.5 | 64.8 | 65.5 | +1.2 [−0.8, +3.5] | −0.8 [−3.5, +2.0] |
| MMLU (570) | 71.2 | 70.4 | 71.4 | −0.9 [−3.0, +1.1] | −1.1 [−3.0, +0.9] |
| HellaSwag (400) | 81.5 | 81.2 | 81.5 | −0.2 [−2.5, +1.8] | −0.2 [−2.5, +2.0] |
| Global‑MMLU‑Lite (400) | 55.2 | 55.2 | 58.2 | 0.0 [−2.2, +2.2] | −3.0 [−5.8, −0.2] |
| WikiText‑2 ppl ↓ | 37.0 | 50.4 | 149.7 | | |
Read honestly:
- Nemotron's post-training transfers onto QAT's 4-bit-lattice weights: the merge recovers most of the QAT→Nemotron
gap wherever Nemotron is ahead (significant on TruthfulQA and MATH), and is statistically indistinguishable from
Nemotron on 9/10 benchmarks while keeping far better raw-text modelling.
- The one significant loss vs Nemotron (multilingual, −3 pp) is inherited from QAT (QAT alone: 55.2, identical) —
the merge itself costs nothing there.
- It does not beat both parents across the board; the GSM8K/MATH margins over Nemotron are inside the noise.
Exceeding both parents would need training, which was out of scope on this CPU-only machine.
Caveats: thinking mode off and greedy decoding (CPU budget), so absolute numbers differ from NVIDIA's
thinking-on card; IFEval sentence/word counting uses a regex splitter (NLTK data unreachable), identically
for every model.
# unfolded (standard)from transformers import AutoModelForImageTextToText, AutoProcessor # AutoProcessor needs torchvision (video)model = AutoModelForImageTextToText.from_pretrained("adityabhushannagar/Gemma4-E4B-Nemotron-Fusion-Unfolded", dtype="bfloat16")
Use the shipped chat_template.jinja (Nemotron's): enable_thinking and conversational_mode work as on the
Nemotron card. fusion/build_final.py --lam 1.0 rebuilds the unfolded checkpoint deterministically from the
three public repos (per-tensor SHA-256 in build_manifest.json).
Layout
fusion/remote_st.py HTTP-range safetensors reader (stream tensors, never the whole file)fusion/materialize.py build local text-decoder dirs / merge store (W_qat + τ, τ as int8)fusion/harness.py CPU bf16 inference, recording hooks, Q4_0 fake-quant, QAT grid recoveryfusion/evals.py benchmark suite (held-out test splits)fusion/screen.py λ screening on practice datafusion/wsp*.py weight-space programmingfusion/fold.py encoder folding + verification + exportfusion/build_final.py deterministic final-checkpoint build from the three public reposresults/ every summary, per-item output, forward-pass log