Model
Table | |
|---|
| Parameters | 7.64B (7.0B text + ~0.45B vision + ~0.18B MTP) |
| Layers | 32 (24 Gated DeltaNet + 8 full attention), from original layers 0–3, 32–43, 48–63 |
| Hidden / MLP | 3584 / 7808 |
| Vocab / context | 248,320 / 262K native (untested after pruning) |
| Weights | bf16, ~15.3 GB |
Pruning details and the shape and block-selection sweeps are on the unhealed repo's card.
Healing so far
Continued pretraining with next-token loss, run as a LoRA relay on free Kaggle 2×T4 GPUs: each session trains a rank-128 LoRA on every language-model projection (attention, linear attention, MLP), merges it, and pushes here. Vision tower, MTP head, embeddings and norms are frozen.
Session 1: 300 steps × 16k tokens = 4.9M tokens, fp16, AdamW 8-bit, LR 2e-4, sequence length 1024.
Data mix: FineWeb-Edu 40%, code (codeparrot-clean) 15%, MiniMax-distill-SFT 15%, Cosmopedia 10%, OpenWebMath 10%, UltraChat 10%.
Held-out cross-entropy (8 × 1024-token blocks per source, fixed across sessions):
Table with columns: Start (unhealed), 1.6M tok, 3.3M tok, 4.9M tok | Start (unhealed) | 1.6M tok | 3.3M tok | 4.9M tok |
|---|
| Web (FineWeb-Edu) | 6.51 | 3.27 | 3.19 | 3.13 |
| Textbook (Cosmopedia) | 7.72 | 2.80 | 2.64 | 2.55 |
| Code | 6.14 | 2.09 | 2.00 | 1.94 |
| Math (OpenWebMath) | 6.00 | 2.96 | 2.89 | 2.84 |
| Chat (UltraChat) | 6.29 | 1.97 | 1.80 | 1.76 |
| Distill* | 4.94 | 1.20 | 1.11 | 1.06 |
| Mean | 6.26 | 2.38 | 2.27 | 2.21 |
*The distill held-out set was drawn from the training split, so it may overlap the training data and overstate the result. From session 2 on, this source is evaluated on its validation split.
Most of the damage was repaired in the first ~1.6M tokens, which suggests the pruned model kept most of its knowledge and mainly needed realignment. Gains are now slowing, as expected. Logs for each session are in heal_log_*.json.
Known issues in this checkpoint
preprocessor_config.json and video_preprocessor_config.json are missing, so image input won't load from this repo yet. Fixed from the next push on.
text_config.dtype says float16 (training ran in fp16); the weights are bf16. Load with an explicit dtype=torch.bfloat16.
- No instruction, reasoning-format, tool-use or vision alignment yet. The model is still being healed, and outputs may be incoherent.
Usage
import torchfrom transformers import AutoModelForMultimodalLM, AutoTokenizer repo = "Sm0r3s/qwen3.8-7b-heal-v2"tok = AutoTokenizer.from_pretrained(repo)model = AutoModelForMultimodalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")
Install flash-linear-attention for fast linear attention. On GPUs without bf16 support (e.g. T4), load with dtype=torch.float16; it gives the same results and is ~3× faster there.
Roadmap
Credits
Base model by the Qwen team (Apache 2.0). Pruning method follows NVIDIA's Minitron work (2407.14679, 2408.11796). Unofficial; not affiliated with Alibaba or Qwen.