What this is
Qwen3.8-27B cut from 27.8B to 7.64B parameters (7.0B text + ~0.45B vision + ~0.18B MTP) by Minitron-style pruning: depth, hidden size and MLP width together. No weights were retrained.
Table with columns: Qwen3.8-27B, This model | Qwen3.8-27B | This model |
|---|
| Parameters | 27.8B | 7.64B |
| Decoder layers | 64 (48 linear + 16 full attention) | 32 (24 linear + 8 full attention) |
| Hidden size | 5120 | 3584 |
| MLP intermediate | 17408 | 7808 |
| Attention heads / KV / head dim | 24 / 4 / 256 | unchanged |
| Linear-attention heads | 16 K / 48 V | unchanged |
| Vocab | 248,320, untied | unchanged |
| Vision encoder | 27 blocks | kept; merger output cut to 3584 |
| MTP head | 1 layer | kept; cut to the same hidden/MLP |
How it was pruned
Importance was scored on the 4-bit base with 144 samples (≤512 tokens) across six domains: English web (FineWeb-Edu), Lithuanian Wikipedia, code, math (GSM8K), chat (UltraChat) and Sm0r3s/MiniMax-distill-SFT. Base mean CE on that set: 1.80.
- Depth. The 64 layers form 16 blocks of 4 (3 Gated DeltaNet + 1 full attention). Blocks were ranked by block influence (1 − cosine similarity between block input and output). The top 8 were kept, always including the first and last block, which preserves the 3:1 attention pattern. Kept blocks 0, 8, 9, 10, 12–15 → original layers 0–3, 32–43, 48–63.
- Hidden size. Residual channels were ranked by mean |RMSNorm output| across all layers (as in Minitron). The top 3584 of 5120 were kept, sliced consistently from embeddings,
lm_head, all norms, every attention / linear-attention / MLP projection, the vision merger output and the MTP head.
- MLP width. In each kept layer, the 7808 most active neurons (of 17408) were kept by mean |activation| at the
down_proj input. MTP MLP neurons were ranked by weight norms, because MTP doesn't run in the standard forward pass.
Full scores are in prune_plan.json and the raw importance tensors in scores.pt. The simulated mean CE of this exact variant before export was 6.34.
Why this shape
Every candidate 7B shape was simulated exactly on the base model before export: dropped hidden channels zeroed and each RMSNorm rescaled by √(h/H). CE averaged over web, Lithuanian wiki, code, math, chat and distill samples:
Table with columns: Shape (all ~7.0B text), Mean CE| Shape (all ~7.0B text) | Mean CE |
|---|
| Base 27B | 1.80 |
| 32 layers, hidden 3584, MLP 7808 | 6.34 |
| 64 layers, hidden 2560, MLP 4224 | 7.44 |
| 40 layers, hidden 3072, MLP 7424 | 9.18 |
| 48 layers, hidden 3072, MLP 4992 | 13.70 |
Earlier sweeps showed that MLP-only cuts are cheap (64 layers with MLP 6912: CE 2.54), while removing depth alone is costly (20 layers: CE 12.3).
Which blocks to keep was tested separately at the same 32-layer shape:
Table with columns: Kept blocks, Mean CE| Kept blocks | Mean CE |
|---|
| 0, 8, 9, 10, 12–15 (this model) | 6.34 |
| 0–5, 14, 15 | 6.96 |
| 0, 8, 10–15 (previous upload) | 7.22 |
| 0–3, 12–15 | 7.70 |
| 0–6, 15 | 7.98 |
| 0–7 (prefix only) | 9.53 |
Keeping the final blocks matters most: a prefix-only model scores far worse.
Status
Usage
import torchfrom transformers import AutoModelForMultimodalLM, AutoTokenizer repo = "Sm0r3s/qwen3.8-7b-pruned-unhealed"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForMultimodalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
The bf16 weights take about 15.3 GB. Install flash-linear-attention and causal-conv1d for fast linear attention; without them transformers uses slower reference kernels.
Limitations
- Unhealed: CE is far above the base model's, and generations will be broken until healing.
- Importance scores come from a 4-bit model and a small calibration set (144 short samples).
- Vision quality, MTP speculative decoding and long context (262K native) are untested after pruning.
Credits
Base model by the Qwen team, Apache 2.0. Method follows NVIDIA's Minitron work (2407.14679, 2408.11796).