GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
sleepyeldrazi
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
sleepyeldrazi
Model Tree
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
Input Modalities
Output Modalities
Supported Functionality
Nemotron 3.5 Lightning is a hybrid Mamba-2 + MoE + Attention model with
hidden=2688, moe_intermediate=1856. Neither is divisible by 256, so
llama.cpp K/IQ quant formats (256-element superblocks) cannot represent the
expert weights — every GGUF quant falls back to block-32 formats (~4.5 bpw
floor) regardless of requested bit width. At 30B that floor is ~18GB; at 20B
it is ~11.5GB. REAP removes the redundant experts so the model fits smaller
quantization budgets without touching the active computation:
REAP prunes the experts your calibration data doesn't use. This model was calibrated on a general-purpose mix so the surviving experts serve general use:
| Source | Share | Content |
|---|---|---|
| instruction (openhermes-style) | 44% | general QA, writing, reasoning, how-to |
| coding (deepseek trajectories + opencode) | 33% | coding problems, agentic code tasks |
| math (metamath-style) | 22% | word problems, arithmetic, reasoning |
Completions were generated by the full model (NVFP4, natural thinking,
per-domain token budgets). If your workload is heavily domain-specific,
re-calibrate on your own data — see reap/.
python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained( "sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B", torch_dtype=torch.bfloat16, trust_remote_code=True)tok = AutoTokenizer.from_pretrained("sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B")For llama.cpp / LM Studio on Apple Silicon, quantize to IQ4_NL (~11.5GB):
bash
llama-quantize model.gguf out-IQ4_NL.gguf IQ4_NLllama-server -m out-IQ4_NL.gguf -c 131072 --reasoning-parser nemotron_v3Context length: -c 131072 (128K) is a safe default for 16GB-VRAM GPUs and
24GB Macs — KV is only ~6KB/token (6 attention layers, 2 KV heads), so 128K
context costs roughly 0.8GB. The model itself supports up to 1M tokens
(256K is NVIDIA's single-GPU deployment length); on laptops the practical
limit at longer contexts is prefill time, not memory — raise -c (e.g.
262144) if your workload needs it.
Sanity-checked with greedy generation on general QA, coding, and math prompts: coherent analysis, correct answers (e.g. math arithmetic verified), no repetition/looping. Full benchmark suites were not run. The closest published pruning analogue (Qwen3-30B-A3B, 128 experts, 8 active) retained ~99.8% of baseline at 50% sparsity.
REAPPruningModifier, plus 3 small patches for the NemotronH architecture (documented in reap/README.md)reap/generate_general_data.pyTo reproduce on your own data: serve the full model, generate your domain corpus
with reap/generate_general_data.py (or your own loader), then run
reap/reap_nemotron.py --data yours.jsonl --sparsity 0.40. Full step-by-step,
patches, and hardware notes: reap/README.md.
OpenMDW License Agreement, version 1.1
(same as the base model). See LICENSE. If you upload derived model files,
carry the same license and attribute NVIDIA for the base weights.
hidden=2688 / moe_intermediate=1856 are not 256-divisible, so llama.cpp GGUF quants floor at block-32 formats (~4.5 bpw); at 20B that is ~11.5GB instead of ~18GB