At a glance

Which variant should I pick?

𓌳 REAP 𓌳 the Experts: Why Pruning Prevails for One-Shot MoE Compression

📄 Paper • 💻 Code

DeepSeek-V3.2-REAP-508B-A37B-NVFP4

REAP-pruned + NVFP4 quantized DeepSeek-V3.2 for efficient deployment on NVIDIA Blackwell (sm120).

This is the first publicly available NVFP4-quantized variant of the 508B-parameter REAP-pruned DeepSeek-V3.2, targeting 8x RTX PRO 6000 Blackwell 96GB deployments via sglang.

📋 Model Specifications

---

🚀 Deploy on sm120 (RTX PRO 6000 Blackwell)

Uses pre-built voipmonitor/sglang:cu130 Docker image with all sm120 patches applied.

bash
Copy code
docker run --gpus all --ipc=host --shm-size=8g --network=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v jit-cache:/cache/jit \
  -e SGLANG_ENABLE_SPEC_V2=True \
  -e SGLANG_ENABLE_JIT_DEEPGEMM=0 \
  -e SGLANG_ENABLE_DEEP_GEMM=0 \
  -e NCCL_IB_DISABLE=1 \
  -e NCCL_P2P_LEVEL=SYS \
  -e NCCL_MIN_NCHANNELS=8 \
  voipmonitor/sglang:cu130 \
  python3 -m sglang.launch_server \
    --model-path 0xSero/DeepSeek-V3.2-508B-NVFP4 \
    --served-model-name deepseek-v32-reap-nvfp4 \
    --tensor-parallel-size 8 \
    --quantization modelopt_fp4 \
    --kv-cache-dtype bf16 \
    --trust-remote-code \
    --attention-backend flashinfer \
    --moe-runner-backend b12x \
    --cuda-graph-max-bs 32 \
    --mem-fraction-static 0.85 \
    --host 0.0.0.0 --port 5000 \
    --disable-custom-all-reduce

Critical flags:

• --kv-cache-dtype bf16 — mandatory; fp8_e4m3 produces garbled output on sm120
• --attention-backend flashinfer — sm120-compatible
• --quantization modelopt_fp4 — sglang's NVFP4 loader for compressed-tensors format
• SGLANG_ENABLE_DEEP_GEMM=0 — DeepGEMM needs WGMMA/TCGEN05 absent on sm120

Memory fit: 288 GB weights + KV cache fits on 8x 96GB (≈768 GB total VRAM).

Will not run on sm_90 (H100): NVFP4 is Blackwell-native. Both vLLM (Marlin FP4 PTX mismatch) and sglang (NotImplementedError: Current platform does not support w4a4 nvfp4 quantization) explicitly block sm_90.

---

🔬 Quantization Method

Produced via AutoRound 0.12.2 layerwise mode on 8x H100 80GB.

Settings

Calibration Dataset

Multi-dataset loading used AutoRound's :concat=true option to pack short samples into full-seqlen sequences.

Wall Time

• Quantization tuning: 19h 38m (61 blocks, ~20 min/block)
• Packing + save: ~7 min (58 safetensors shards, 288 GB)
• Total: ~19.7 hours on 8x H100 80GB

Quality Characteristics

Layer-level loss trajectory (iter 0 → final):

Weight-validity check (CPU dequant, pre-upload):

• Cosine similarity vs BF16 source: 0.995+ across all tested layers
• Relative MAE: ~9% uniformly (typical NVFP4 reconstruction error)

---

📊 Benchmarks

Pending. Run on 8x RTX PRO 6000 sm120 and report:

Expected ranges (based on GLM-5.1-555B-A14B-REAP-NVFP4 precedent):

• MMLU: 73-79% (BF16 base ~75-80%, −1 to −2 pp for NVFP4)
• GSM8K: 78-88% (BF16 base ~80-90%)
• Decode throughput: 50-70 tok/s @ conc=1, 120-180 tok/s @ conc=4

---

🧾 Provenance

---

License & citation

License inherited from the base model.

bibtex
Copy code
@misc{lasby2025reap,
  title  = {REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression},
  author = {Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa},
  year   = {2025}, eprint = {2510.13999}, archivePrefix = {arXiv}
}

Sponsors

Made possible by NVIDIA · TNG Technology · Lambda · Prime Intellect · Hot Aisle.