erokhins

Qwen3.6-27B-FFN-pruned-25

README

License: apache-2.0

Method: drift-guided structured pruning

Qwen3.8-27B is a continued-training descendant of Qwen3.6-27B: every FFN channel sits at the same index in both, and no channel was replaced (minimum same-index cosine across the whole model is ~0.52). That makes per-channel cross-version drift measurable and usable as a pruning signal.

For each FFN channel c in layer L:

markdown
score(L,c) = min( cos(W_gate³·⁶[c,:], W_gate³·⁸[c,:]),
                  cos(W_up³·⁶[c,:],   W_up³·⁸[c,:]),
                  cos(W_down³·⁶[:,c], W_down³·⁸[:,c]) )

Worst-of-three: a channel only counts as stable if its gate row, up row and down column all survived continued training.

Selection (p2030r25)

Protect the most stable channels, with a depth ramp: top 20% of layer 0 rising linearly to top 30% of layer 63 (25% of all channels protected). The ramp follows the measured stability gradient — mean drift cosine rises from ~0.72 in early layers to ~0.91 in the last, so late layers hold more genuinely converged channels.
Cut 25% of each layer at random from the unprotected pool. Removing a channel means deleting its gate row, up row and down column together, which is exact: the SwiGLU intermediate dimension has no cross-channel interaction, so channels can be permuted and truncated without changing the function of what remains.

Why random inside the pool, rather than "cut the most-drifted"? Controlled experiments found drift to be a weak selection signal — at equal volume, cutting the most-drifted channels performed no better than random. But it is a strong protection signal: cutting the most stable channels instead caused severe fluency collapse at just 30%. Stability marks converged, load-bearing machinery. So the protection set does the work and random picking fills the quota.

The MTP head (mtp.layers.0) carries its own 17408-wide SwiGLU, so it is pruned the same way and to the same width (its channels scored a mean drift cosine of 0.850, comparable to the main layers; protection set to 30%, matching the ramp's end since the head sits after the last layer). mtp.fc, the MTP norms and its attention are untouched.

Usage

Standard transformers / mlx-vlm loading — the config declares the reduced intermediate_size, so no custom code is needed.

python
from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
    "erokhins/Qwen3.6-27B-FFN-pruned-25", dtype="bfloat16", device_map="auto")
processor = AutoProcessor.from_pretrained("erokhins/Qwen3.6-27B-FFN-pruned-25")

Run it with thinking disabled (enable_thinking: false in the chat template). In non-thinking mode it answers cleanly and concisely. With thinking enabled it tends to over-deliberate — note that unpruned Qwen3.6 is also verbose in thinking mode, so pruning amplifies an existing trait rather than creating it.

Limitations

Evaluated only by smoke testing, not by perplexity or benchmark suites. Treat quality claims as provisional.
Pruning damage in this family shows up first in the direct-answer pathway and as occasional dropped short tokens, not as gibberish — watch for terse or empty responses rather than incoherence.
Aggressive pruning of the GDN mixers was tried separately and damaged real usage even at 12.5%; this model leaves them alone. FFN-only is the safe cut here.
Not instruction-tuned or fine-tuned after pruning. A least-squares reconstruction of W_down on calibration activations would likely recover some quality and is the obvious next step.

Reproducing

Scripts, masks and a full experiment write-up (including the pruning-cliff study: the unprotected cliff sits near 50% of FFN channels, ~57% with flat protection, ~60% with this depth-adaptive scheme) are in the project repository.

bash
# 1. per-channel drift scores from the two parent checkpoints
uv run compare_36_38.py
# 2. the p2030r25 mask
uv run make_hybrid_mask.py --scheme protect --frac 25 \
    --floor-ramp 20 30 --pick random --uniform-count --seed 25 \
    --out null_mask_p2030r25.npy
# 4. extend the mask with a row for the MTP head's FFN, then build & truncate
uv run add_mtp_to_mask.py null_mask_p2030r25.npy null_mask_p2030r25_mtp.npy 30 25
COEFF1=1.0 uv run build_null_merge.py zeros-27B null_mask_p2030r25_mtp.npy
uv run truncate_model.py zeros-27B Qwen3.6-27B-FFN-pruned-25 \
    null_mask_p2030r25_mtp.npy 13056

Attribution

Derived from Qwen/Qwen3.6-27B (Apache-2.0); channel selection additionally used Qwen/Qwen3.8-27B for comparison only — none of its weights are present in this model. Released under Apache-2.0, as the base model.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

erokhins

Model Tree

Base

Qwen/Qwen3.6-27B

Fine-tuned

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality