At a glance
Table | |
|---|
| Base model | PrimeIntellect/INTELLECT-3 |
| Format | BF16 |
| Total params | 57B |
| Active / token | — |
| Experts / layer | 64 |
| Layers | 46 |
| Hidden size | 4096 |
| Context | 131,072 |
| On-disk size | 114 GB |
50% expert-pruned version of PrimeIntellect/INTELLECT-3 using Cerebras REAP (Router-weighted Expert Activation Pruning).
Model Details
Table with columns: Property, Value| Property | Value |
|---|
| Base Model | PrimeIntellect/INTELLECT-3 (248B MoE) |
| Architecture | GLM-4 MoE (glm4_moe) |
| Compression | 50% (64 experts pruned) |
| Remaining Experts | 64 per layer |
| Parameters | ~124B |
| Format | BF16 SafeTensors |
| Size | 107 GB |
REAP Configuration
dataset: 0xSero/glm47-reap-calibration-v2
samples: 1360
- evol-codealpaca-v1: 700 (code generation)
- xlam-function-calling-60k: 330 (function calling)
- SWE-smith-trajectories: 330 (agentic multi-turn)
distance_measure: angular
seed: 42
model_max_length: 2048
compression_ratio: 0.50
prune_method: reap
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"0xSero/INTELLECT-3-57B",
torch_dtype="auto",
device_map="auto",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("0xSero/INTELLECT-3-57B", trust_remote_code=True)
messages = [{"role": "user", "content": "Write a Python function to calculate fibonacci numbers"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Table with columns: Model, Compression, Format, Size| Model | Compression | Format | Size |
|---|
| INTELLECT-3-REAP-50 | 50% | BF16 | 107GB |
| INTELLECT-3-REAP-50-W4A16 | 50% | W4A16 GPTQ | ~30GB (coming soon) |
License & citation
License inherited from the base model.
@misc{lasby2025reap,
title = {REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression},
author = {Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa},
year = {2025}, eprint = {2510.13999}, archivePrefix = {arXiv}
}
Made possible by NVIDIA · TNG Technology · Lambda · Prime Intellect · Hot Aisle.