ikimyaii

llama-7b-cws-80pct

Model details

Table

Base model	`openlm-research/open_llama_7b`
Method	CWS (block, B = 128)
Target / achieved sparsity	80% / 79.69%
Weight correction	OBS (second-order)
Retraining	none

What is pruned: every nn.Linear in the 32 decoder layers (attention q/k/v/o and MLP gate/up/down) plus lm_head, 225 layers in total (see pruning_summary.json). Embeddings and norms stay dense.

Evaluation

Table with columns: Metric, This model, Dense
Metric	This model	Dense
WikiText-2 PPL ↓	948.5	8.64
ARC-Easy (acc_norm)	0.252	0.678
ARC-Challenge (acc_norm)	0.282	0.385
HellaSwag (acc_norm)	0.261	0.694
PIQA (acc_norm)	0.516	0.763
WinoGrande (acc)	0.500

Zero-shot via lm-evaluation-harness; ARC/HellaSwag/PIQA acc_norm, WinoGrande acc. Raw results: results/benchmark_llama_obs_cancel_block_80pct.json.

WikiText-2 PPL of all five methods at 80% (dense 8.64):

Table with columns: Method, WikiText-2 PPL ↓
Method	WikiText-2 PPL ↓
CWS (this model)	948.5
SparseGPT	1,103.6
Wanda	1,252.3
RIA	1,609.5
AWP	230.10

These are the project's results at the time the checkpoints were uploaded (2026-05-11; results/sparsity_sweep_llama.json @ 6e144ae), with dense PPL 8.64. The repository README now reports these models under a 2048-token protocol (dense 6.61). The README's CWS numbers also come from a later sequential-calibration version of CWS, which is not this checkpoint, and are better (e.g. 8.24 at 50%).

About CWS

CWS (Correlation-aware Weight Sparsity) is a post-training, one-shot pruning method that extends SparseGPT's Optimal Brain Surgeon (OBS) weight correction with a full-Hessian selection criterion.

Per-weight saliency scores (magnitude, Wanda, RIA, SparseGPT's diagonal score) look at each weight in isolation. They miss cancellation groups: correlated input channels whose opposite-sign weights partly cancel in the layer output, so removing them together costs less than their individual scores predict. For each output row, CWS greedily picks the weight with the smallest marginal cost δ(j | S) = r_j² / d_j, where r_j and d_j come from the evolving inverse-Hessian factor (one rank-1 Schur-complement downdate per pick). It then applies exact OBS corrections to the surviving weights. With H = (2/N) XᵀX, the first pick is exactly SparseGPT's choice; later picks account for the cross-weight interactions that a diagonal score ignores.

Two selection scopes appear in this collection:

Block CWS (default): greedy selection is confined to 128-column blocks, and corrections are applied block by block. This is stable on wide layers.
Global CWS (ablation): the whole row is one block (block size → ∞). It captures cross-block cancellation, but it is more prone to numerical drift on large layers.
Method and reference implementation: CWS
Experiments, baselines and raw results for these checkpoints: Sparse-LLMs-on-Loihi
Follow-up work that swaps CWS's layer-wise reconstruction objective for the real next-token loss: Weight-Sparsity-Using-Real-Loss

Usage

python

import torchfrom transformers import AutoModelForCausalLM, LlamaTokenizer model_id = "ikimyaii/llama-7b-cws-80pct"tokenizer = LlamaTokenizer.from_pretrained(model_id)model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto") lin = model.model.layers[0].mlp.down_projprint(f"sparsity: {(lin.weight == 0).float().mean():.3f}")

Limitations

These are research checkpoints for studying pruning. They are not instruction-tuned or safety-tuned, and they inherit the base model's biases.
Weights are stored densely, with zeros at the pruned positions. Sparsity alone gives no speedup without a sparse kernel or event-driven hardware.
At 70–80% sparsity this model does not produce usable text. Zero-shot accuracy is near chance.

Citation

The CWS paper is under anonymous review; please cite the code repository for now.

bibtex

@misc{cws2026,  title  = {Semi-Structured Correlation-aware Weight Sparsity (CWS) for Efficient LLM Inference},  year   = {2026},  note   = {Under review. Code: https://github.com/KimiaGholami/CWS}} @inproceedings{frantar2023sparsegpt,  title     = {SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot},  author    = {Frantar, Elias and Alistarh, Dan},  booktitle = {International Conference on Machine Learning},  year      = {2023}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

ikimyaii

Model Tree

Base

openlm-research/open_llama_7b

Fine-tuned

this model

Input Modalities

Text

Output Modalities