At a glance
Which variant should I pick?
Table with columns: Variant, Format, Link| Variant | Format | Link |
|---|
Nemotron-3-Super-64B (this) | BF16 | link |
Nemotron-3-Super-64B-W4A16 | W4A16 | link |
Nemotron-3-Super-92B | BF16 | link |
This repo is a draft REAP-derived checkpoint based on nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16.
Provenance
Research References
Pruning details
- Experts per MoE layer in upstream base:
512
- Experts retained per layer in this variant:
256
- Experts pruned per layer in this variant:
256
- Expected safetensor shard count in this draft repo:
3
- Source merged observation workflow:
nemotron_super_merged_long50_short15120_v2
Method summary
The pruning signal comes from layerwise REAP observations collected over a mixed calibration corpus dominated by a personal AI-session history plus a bounded public augmentation slice.
Validated observation lanes used in the merged signal:
nemotron_super_long50_16k_v3
- longest personal trajectories first
50 trajectories- capped at
16384 tokens each
nemotron_super_short_mix_15120_t1024_b8192_v4
15000 short personal prompts plus 120 bounded public prompts- capped at
1024 tokens each
- packed under a safe
8192 token batch budget
Model facts from the merged observation lane:
- runtime architecture class:
NemotronHForCausalLM
- total blocks:
88
- MoE blocks:
40
- Mamba blocks:
40
- attention blocks:
8
- routed experts per token:
22
Intended use
This draft checkpoint is published for research into expert activation structure, residency planning, CPU offloading, and prompt-conditioned expert selection. It is not a production claim and it is not an NVIDIA release.
Draft caveats
- This is a draft derived checkpoint.
- We have not yet completed a full serving benchmark and quality benchmark campaign for this release on Hugging Face.
- The repo preserves provenance back to the upstream NVIDIA release and should be evaluated in that context.
License and terms
Distribution of this derived checkpoint is intended to comply with the NVIDIA Open Model License included in LICENSE.txt. The required attribution notice is included in NOTICE.
License & citation
License inherited from the base model.
@misc{lasby2025reap,
title = {REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression},
author = {Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa},
year = {2025}, eprint = {2510.13999}, archivePrefix = {arXiv}
}
Made possible by NVIDIA · TNG Technology · Lambda · Prime Intellect · Hot Aisle.