cosmos1030

gmp-kd3e-1-s70pct-lr1e-4_20260901_080954

README

License: apache-2.0

⚠️ Which run is this?

Five Qwen3-4B 70% ablation arms were uploaded with near-identical repo names differing only by timestamp. This repo is the headline arm — A0, the full method at δ = 0.02, the best-scoring 70% cell:

Table with columns: repo (timestamp suffix), arm, avg5
repo (timestamp suffix)	arm	avg5
`…_20260901_080954` ← this repo	A0 — SCOUT, δ = 0.02 (full method)	45.58
`…_20260903_081638`	B1 — w/o rollout refresh (frozen dense pool)	43.97
`…_20260903_144159`	A2m — schedule k-rule, pace-matched	42.75
`…_20260903_072918`	A3 — w/o trust region (δ = ∞)	41.87
`…_20260903_072500`	A1 — SCOUT, δ = 0.01	41.20

All rows: Qwen3-4B, 70% unstructured, lr 1e-4, 2048 steps, everything except the named component held identical. Single seed (42), one run per cell.

Results

Evaluated with lighteval (quick profile), 5 tasks:

Table with columns: Task, Metric, Score
Task	Metric	Score
MATH-500	accuracy †	72.40
GSM8K	`extractive_match`	71.49
GPQA-Diamond	`gpqa_pass@k:k=1`	35.86
IFEval	`prompt_level_strict_acc`	39.93
LiveCodeBench (codegen)

† MATH-500 is quoted from the source ablation table; the other four scores were re-read from this repo's own lighteval result JSONs and match that table exactly.

Per-task raw lighteval outputs (results/*.json + details/*.parquet) are included in this repo under lighteval_bench/.

Method

Growth is driven entirely by PGD — there is no separate schedule- or cubic-driven growth mechanism (--gmp_tr_enabled=false, --gmp_pruning_end_ratio=0.0). At every 8th step the mask's desired state is the top-k-by-importance projection at the final target sparsity, and a bisection accepts the largest swap set k whose measured self-KL displacement stays within the trust region δ:

Saliency: Fisher-weighted magnitude (fisher_factor · w², Fisher diagonal from Adam's exp_avg_sq), global scope across all masked linears.
Trust region: δ = 0.02, with the self-KL measured on a 4-sequence calibration batch drawn from the current rollout pool and refreshed every 32 steps.
Asymmetric prune/revive: revive saturates at min(k, revive_cand) while pruning continues to k, so accepted swaps are net-pruning until the target is met, after which the same rule degrades to pure maintenance.
Recovery objective: NTP + offline KD + on-policy KD, weighted 0.33/0.33/0.33. The on-policy term distills the dense teacher on the student's own rollouts, generated by an in-process vLLM engine (512 new tokens, temperature 0.6) and refreshed every 32 steps (64 refreshes over the run).

Growth is fast, recovery is long: the target 70% was reached at step 232 of 2048 (11.3%); the remaining 88.7% of training is recovery and mask maintenance.

Training configuration

Table

Base / teacher	`Qwen/Qwen3-4B` (dense, frozen)
Sparsity	70% unstructured, global scope (final measured: 0.7000)
Steps	2048 (batch 1 × grad-accum 8, seq len 8192)
LR	1e-4, cosine, 256-step linear warmup
Mask / PGD cadence	mask_interval 32, pgd_interval 8
Trust region δ	0.02 (`--gmp_pgd_kl_budget`), calib size 4
Saliency	`fisher`, Fisher EMA β = 0.999

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "cosmos1030/gmp-kd3e-1-s70pct-lr1e-4_20260901_080954"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")

msgs = [{"role": "user", "content": "Solve: what is the 12th Fibonacci number?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0][ids.shape[-1]:], skip_special_tokens=True))

Limitations

No speedup on its own. Sparsity is unstructured and the weights are stored as ordinary dense safetensors containing zeros. Wall-clock or memory gains require sparse kernels or a structured (e.g. 2:4) variant; loading this model as-is costs the same as the dense base model.
Single seed, single run per cell. All ablation numbers above come from one run at seed 42. The source ablation package states its own resolution as roughly ±1 point, so differences smaller than that are not resolved — the B1 row in particular sits inside that band at other sparsities.
Evaluation protocol. Numbers come from one lighteval quick-profile configuration and are directly comparable within this ablation family; they are not necessarily comparable to numbers produced under other papers' harnesses, prompts, or sampling settings.
Inherits the capabilities, biases, and failure modes of Qwen/Qwen3-4B. No dense-baseline row is included here, so the tables above quantify differences between pruning arms, not the cost of pruning relative to the dense model. Absolute code-generation ability is low at this sparsity (8.21 on LiveCodeBench).

Provenance

Training log: logs/queue_gmp_pgd_grow_to_target_4b/s70_klb0.02.log
W&B run: r5j1uw8d (project reasoning_qwen3_4b_nostrip8192)
Launcher: b200_scripts/gmp_pgd_grow_to_target_qwen3_4b.sh 0.7 0.02 512 32 cosine 2048 1e-4 $OT3_DATA 8192 true reasoning_qwen3_4b_nostrip8192 fisher global 0.33,0.33,0.33 32 0 4 8 0.15
Trained 2026-09-01, uploaded 2026-09-01 08:10 KST

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

cosmos1030

Model Tree

Base

Qwen/Qwen3-4B

Fine-tuned

this model

Input Modalities

Text

Output Modalities